Redis optimization

Retroactive User Recognition is using Redis to store all anonymous incoming events. Usually, the volume of such event can be pretty huge, so Redis can take a lot of RAM (note: Redis keeps all data in RAM, disk is used only for persistence)

Approximate of event in Redis is 1KB. So if the project processes requests per second, and let's say 80% of them are anonymous, the daily consumption will be:

By default, Jitsu will mark users not active for 7 days for eviction, so total Redis consumption will be around ~5G (number of inactivity days is controlled by USER_RECOGNITION_TTL_MINUES)

Redis keys for user recognition format is anonymous_events:destination_id#${destinationID}:anonymous_id#${userAnonymousID}. Example: anonymous_events:destination_id#my_postgres:anonymous_id#bqsyuwusxh. All events generated by this user will be stored under this key is list

The most reliable way to handle the RAM problem is having a dedicated Redis instance for user recognition with memory limit and keys eviction configuration. You can configure it with the following steps:

1. Configure separate Redis in Jitsu Server configuration file (eventnative.yaml) with anonymous events TTL (default value is 7 days):#

users_recognition:
  enabled: true
  identification_nodes:
    - /user/id
    - /user/email
  redis:
    host: redis_host
    port: 6379
    password: secret_password
    ttl_minutes: #Optional. Anonymous events TTL in minutes. Default value: 10080 (7 days)
      anonymous_events: 10080 #7 days

Alternatively, you can configure dedicated redis with env variable (USER_RECOGNITION_REDIS_URL=redis://:password@host:port) and TTL with USER_RECOGNITION_TTL_MINUTES

TTL (time to live) is applied to Redis keys. It means that events from anonymous users that no longer visit your web pages will be deleted from Redis at the expiration time. Under the hood Jitsu uses expire command after saving every anonymous event JSON. Since Redis anonymous events key contains user anonymous ID, Redis will automatically delete key if there was no events from a certain anonymous user 7 days (TTL value).

2. Specify Redis memory limit and eviction policy by adding the following configuration into the redis.conf file:#

Or use command CONFIG SET with redis-cli:

After this configuration, user recognition Redis will automatically rewrite older keys when RAM is almost full.

Add Redis compression configuration, so all Redis hash tables with length > 1 (anonymous with more than 1 event inside 1 key) will be converted to the Redis ziplists and compressed. Compression rate is about 8x-10x. It saves about 800-1000% of RAM.

Configuration:

Or use command CONFIG SET with redis-cli:

You can check how much memory any key consumes with redis-cli command:

If you can't change the redis configuration, you can enable GZIP compression on the Jitsu side. It will approximately reduce Redis memory consumption by 30%. Just add users_recognition.compression: 'gzip' to the Jitsu Server configuration (eventnative.yaml):

users_recognition:
  enabled: true
  compression: 'gzip'
  identification_nodes:
    - /user/id
    - /user/email
  redis:
    host: redis_host
    port: 6379
    password: secret_password
    ttl_minutes: #Optional. Anonymous events TTL in minutes. Default value: 10080 (7 days)
      anonymous_events: 10080 #7 days