HAYASHIER.COM - Private Page
Memcached 公式ドキュメント まとめ

memcached のWikiを一読してまとめてみました (2018/05/13 時点)

Memcached

別途、Memcachedの機能のまとめ記事等は以下のURLを参照いただければと存じます。 Dive Deep Memcached ~ 入門から実装の確認まで ~ Redis / Memcached Source Code Reading – Overview – Dive Deep Memcached ~ SETコマンド実行時の動作 ~

About

Overview

Release Notes

Communication, Mailing List

Ancient ML Archives

IRC: - irc.freenode.net #memcached

Want to contribute to this wiki? Submit a PR!

Getting Started

Installing

# パッケージから取得
## Debian, Ubuntu
$ apt-get install libevent-dev
$ apt-get install memcached

## Redhat/Fedora
$ yum install libevent-devel
$ yum install memcached

# ソースからビルド
wget https://memcached.org/latest
[you might need to rename the file]
tar -zxf memcached-1.x.x.tar.gz
cd memcached-1.x.x
./configure --prefix=/usr/local/memcached
make && make test && sudo make install

Tutorial (in fun story form!)

$MEMCACHE_SERVERS = array(
    "10.1.1.1", //web1
    "10.1.1.2", //web2
    "10.1.1.3", //web3
);

:
:

$memcache = new Memcache();
foreach($MEMCACHE_SERVERS as $server){
    $memcache->addServer ( $server );
}

:
:

$huge_data_for_front_page = $memcache->get("huge_data_for_front_page");
if($huge_data_for_front_page === false){
    $huge_data_for_front_page = array();
    $sql = "SELECT * FROM hugetable WHERE timestamp > lastweek ORDER BY timestamp ASC LIMIT 50000";
    $res = mysql_query($sql, $mysql_connection);
    while($rec = mysql_fetch_assoc($res)){
        $huge_data_for_front_page[] = $rec;
    }
    // cache for 10 minutes
    $memcache->set("huge_data_for_front_page", $huge_data_for_front_page, 0, 600);
}

// use $huge_data_for_front_page how you please
### Hardware Recommendations

Hardware Recommendations

Configuration

Configuring a Server

$ echo "stats settings" | nc localhost 11211
STAT maxbytes 67108864
STAT maxconns 1024
STAT tcpport 11211
STAT udpport 11211
STAT inter NULL
STAT verbosity 0
STAT oldest 0
STAT evictions on
STAT domain_socket NULL
STAT umask 700
STAT growth_factor 1.25
STAT chunk_size 48
STAT num_threads 4
STAT stat_key_prefix :
STAT detail_enabled no
STAT reqs_per_event 20
STAT cas_enabled yes
STAT tcp_backlog 1024
STAT binding_protocol auto-negotiate
STAT auth_enabled_sasl no
STAT item_size_max 1048576
END

Configuring a Client

External flash storage

               THREADS            +            STRUCTURES
 +----------------------+         |
 |-----------|          |         |     +----------+
+------+  +--v-----+ +--v-----+   |     |hash table|
|listen|  |worker 1| |worker 2|   |     +----------+
+------+  +--------+ +--------+   |
                        +--------->     +---+
                                  |     |LRU|
+--------------+                  |     +---+
|lru maintainer+------------------>
+--------------+                  |     +--------------+
                                  |     |slab allocator|
+-----------+                     |     +--------------+
|lru crawler+--------------------->
+-----------+                     |
                                  |
+---------------+                 |
|slab page mover+----------------->
+---------------+                 |
                                  +

フラッシュの場合は、storage が LRU maintainer と似た働きをする。compaction も IO に作用する。

+--------+
|LRU tail|
+--------+
  |
  | * Is the item large?               +------>
  |                                    |
  | * Is memory low?                   | Once written to buffer:
  |                                    |
  | * Is a write buffer available?     | * Allocate new small item
  |                                    |
  |   +------------+                   | * Copy key+metadata to new item
  +--->write buffer+-------------------+ * Also contains pointer to flash:
      +------------+                       [page, offset, version]

    * IF the write buffer was full:      * Replace original with small item

     +------------+     +---------+
     |write buffer+----->IO thread|
     +------------+     +---------+

    * All writes wait during flush
OPEN      [BUCKETS]     FULL

          +-------+
[P9]<-----+DEFAULT+---->[P0,P1,P2,P5,P6]
          +-------+

          +---------+
[P10]<----+COMPACTED+-->[P3,P4]
          +---------+

          +-------+
[P11]<----+LOW TTL+---->[P7,P8]
          +-------+


FREE: [P12,P13,...]

Usage

Clients

API Commands

Common Client Features

特徴貼り付け!!!!

Programming With Memcached Basics

# perl
my $memclient = Cache::Memcached->new({ servers => [ '10.0.0.10:11211', '10.0.0.11:11211' ]});
# pseudocode
memcli = new Memcache
memcli:add_server('10.0.0.10:11211')
# Don't load little bobby tables
sql = "SELECT * FROM user WHERE user_id = ?"
key = 'SQL:' . user_id . ':' . md5sum(sql)
 # We check if the value is 'defined', since '0' or 'FALSE' # can be
 # legitimate values!
if (defined result = memcli:get(key)) {
    return result
} else {
    handler = run_sql(sql, user_id)
    # Often what you get back when executing SQL is a special handler
    # object. You can't directly cache this. Stick to strings, arrays,
    # and hashes/dictionaries/tables
    rows_array = handler:turn_into_an_array
    # Cache it for five minutes
    memcli:set(key, rows_array, 5 * 60)
    return rows_array
}
sql1 = "SELECT * FROM user WHERE user_id = ?"
sql2 = "SELECT * FROM user_preferences WHERE user_id = ?"
key  = 'SQL:' . user_id . ':' . md5sum(sql1 . sql2)
if (defined result = memcli:get(key)) {
    return result
} else {
    # Remember to add error handling, kids ;)
    handler = run_sql(sql1, user_id)
    t[info] = handler:turn_into_an_array
    handler = run_sql(sql2, user_id)
    t[pref] = handler:turn_into_an_array
    # Client will magically take this hash/table/dict/etc
    # and serialize it for us.
    memcli:set(key, t, 5 * 60)
    return t
}
 # Lets generate a bio page!
user          = fetch_user_info(user_id)
bio_template  = fetch_biotheme_for(user_id)
page_template = fetch_page_theme
pagedata      = fetch_page_data

bio_fragment = apply_template(bio_template, user)
page         = apply_template(page_template, bio_fragment)
print "Content-Type: text/html", page
key = 'FRAG-BIO:' . user_id 
if (result = memcli:get(key)) {
    return result
} else {
    user         = fetch_user_info(user_id)
    bio_template = fetch_biotheme_for(user_id)
    bio_fragment = apply_template(bio_template, user)
    memcli:set(key, bio_fragment, 5 * 15)
    return bio_fragment
}

Memcached Usage FAQ

HOWTO’s and Tricks

user_prefix = memcli:get('user_namespace:' . user_id)
bio_data    = memcli:get(user_prefix . user_id . 'bio')

Unixタイムを利用

# Namespace management, basic fetch.
key = 'namespace:' . user_id
namespace = memcli:get(key)
if (!namespace) {
    namespace = time()
    if (! memcli:add(key, namespace)) {
        # lost the race.
        namespace = memcli:get(key)
        # Could re-test it and jump to the start of the loop, hard fail, etc.
    }
    # Send back the new namespace.
    return namespace
}

Invalidation

key = 'namespace:' . user_id
if (! memcli:incr(key, 1)) {
    # Increment failed! Key must not exist.
    memcli:add(key, time())
}
if (data = memcli:get('helloworld')) {
    # Yay stuff!
}

Memcached Internals for End Users

$ ./memcached -vv
slab class   1: chunk size        80 perslab   13107
slab class   2: chunk size       104 perslab   10082
slab class   3: chunk size       136 perslab    7710
slab class   4: chunk size       176 perslab    5957
slab class   5: chunk size       224 perslab    4681
slab class   6: chunk size       280 perslab    3744
slab class   7: chunk size       352 perslab    2978
slab class   8: chunk size       440 perslab    2383
slab class   9: chunk size       552 perslab    1899
slab class  10: chunk size       696 perslab    1506
[...etc...]

Maintenance

Maintaining a Server

Maintaining a Cluster

What Performance To Expect

Troubleshooting Timeouts

Development

Development Repos

Protocols

Protocol

BinaryProtocolRevamped

     Byte/     0       |       1       |       2       |       3       |
        /              |               |               |               |
       |0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7|
       +---------------+---------------+---------------+---------------+
      0/ HEADER                                                        /
       /                                                               /
       /                                                               /
       /                                                               /
       +---------------+---------------+---------------+---------------+
     24/ COMMAND-SPECIFIC EXTRAS (as needed)                           /
      +/  (note length in the extras length header field)              /
       +---------------+---------------+---------------+---------------+
      m/ Key (as needed)                                               /
      +/  (note length in key length header field)                     /
       +---------------+---------------+---------------+---------------+
      n/ Value (as needed)                                             /
      +/  (note length is total body length header field, minus        /
      +/   sum of the extras and key length body fields)               /
       +---------------+---------------+---------------+---------------+
       Total 24 bytes
     Byte/     0       |       1       |       2       |       3       |
        /              |               |               |               |
       |0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7|
       +---------------+---------------+---------------+---------------+
      0| Magic         | Opcode        | Key length                    |
       +---------------+---------------+---------------+---------------+
      4| Extras length | Data type     | vbucket id                    |
       +---------------+---------------+---------------+---------------+
      8| Total body length                                             |
       +---------------+---------------+---------------+---------------+
     12| Opaque                                                        |
       +---------------+---------------+---------------+---------------+
     16| CAS                                                           |
       |                                                               |
       +---------------+---------------+---------------+---------------+
       Total 24 bytes
     Byte/     0       |       1       |       2       |       3       |
        /              |               |               |               |
       |0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7|
       +---------------+---------------+---------------+---------------+
      0| Magic         | Opcode        | Key Length                    |
       +---------------+---------------+---------------+---------------+
      4| Extras length | Data type     | Status                        |
       +---------------+---------------+---------------+---------------+
      8| Total body length                                             |
       +---------------+---------------+---------------+---------------+
     12| Opaque                                                        |
       +---------------+---------------+---------------+---------------+
     16| CAS                                                           |
       |                                                               |
       +---------------+---------------+---------------+---------------+
       Total 24 bytes

SASLAuthProtocol

RangeOps

Get     0x30    rget
Set     0x31    rset
Quiet Set   0x32    
Append  0x33    rappend
Quiet Append    0x34    
Prepend     0x35    rprepend
Quiet Prepend   0x36    
Delete  0x37    rdelete
Quiet Delete    0x38    
Incr    0x39    rincr
Quiet Incr  0x3a    
Decr    0x3b    rdecr
Quiet Decr  0x3c