Search

Todd Rodzen

Agile Application Development

Tag

redis

UUID vs Auto Increment

What is the best method to create a key in today’s advanced Javascript node.js style applications? Do you rely on the old tried and true method to use auto-increment on the database primary key? or is a UUID better? One thought to help answer that; is a sequential key useful? Especially in the situation where the unique key may start out as only in the application or only on the client session store (ie. Redis key memory store.) In this situation, a sequential key is not useful and creating the auto increment key takes an additional step using INCR on Redis or INSERT on MySQL which could also create an unnecessary round trip to your database.

On the other hand, the UUID v4 implementation which creates a unique randomized UUID may appear to be a CPU time slice consuming operation, but one stackoverflow user did some testing, as posted here,:

uuid.png

You can see the green and yellow lines of increased connections. As connections increase AUTO INCR method creates an increased latency; while the UUID is a steady same or lower process time slice.

I didn’t come up with this one but found it may possibly be the smallest UUID v4 generator code.

exports.uuid = function b(a){return a?(a^Math.random()*16>>a/4).
   toString(16):([1e7]+-1e3+-4e3+-8e3+-1e11).replace(/[018]/g,b)}

There is no one answer. It’s always good to have multiple available methods but be sure to consider the uses and weigh the options.

Redis Cluster with Passwords

Do a little work with Redis Clusters and you will see in multiple places developers trying to get Redis node instances with passwords to work in a Redis cluster environment. The fact is it’s not supported. It’s not an option for good reason. There is a second back data channel that essentially makes the password AUTH meaningless. On top of that, passwords on a memory key store is well, meaningless for a good hacker. If you can throw thousands of passwords at the instance in ONE SECOND then the brute force hack is pretty easy. Maybe future versions of Redis will start to take password retries into consideration.

On the other hand, there are good reasons for a password on any service. A couple reasons come to mind: 1. You simply want to stop inadvertent prying eyes, such as an employee within the company that has access to the machine and the redis-cli command tool. 2. Maybe you post your passwords on a sticky note next to the computer room monitor so the password itself is not a concern but the person that has access to the machine but not the computer room should be staying out of the data. 3. You have multiple people that work on the machine and you want to protect your instance so a co-developer doesn’t accidently access and delete your Redis node. The list could probably go on much longer. One thing for sure, even if you assign a password to your Redis instance if you open the port up to the public you are opening yourself up to a hack. On the other hand, if it has a password and you are only using it for testing and development, maybe it’s not a big deal.  The better option is to use SSH to tunnel to your Redis server over the internet. That has its own issues.

One reason I chose to setup a cluster with AUTH passwords is I wanted to build apps on my laptop running locally on my laptop node.js server. I want the app to connect to my remote MYSQL development/production database and the same type of situation for the memory key store. That way, in theory, you can develop and test a version on the laptop. You can push it to the development EC2 server without any code changes and it should also work because it would be using the same MYSQL database (connecting to hostname mysql.mydomain.com) and Redis Cluster connection to a hostname redis.mydomain.com. (it won’t be using a local cluster or node on my laptop during development.)

With a Redis cluster environment, there is a back channel communications port for the cluster for each Redis node instance. The communications port is the node’s port with a 1 in front of the number. So if you have a node sitting on port 6101 there is also a back channel cluster communications port of 16101. We don’t use it. It’s only used by the Redis server. So In my situation above, I will not open the communications port to the public.

six-node-redis-cluster

Furthermore, why go to all this trouble if you are just working on a development application. Well, in theory, your development application will soon be a minimum viable product (MVP) and that won’t happen, or be much more difficult later, if you develop an application with code using a single memory store environment and then have to transition to a whole new API client for production. It’s better to develop an application once the right way. If you are developing an application that will have widespread use you know the cluster environment is needed. It may be a question of your development process and some won’t want to take this approach. But if you develop with a single node you might expect a multi-stage redevelopment as clusters are needed down the road and that adds a few steps.

With my development scenario, I am using the same server for all six nodes with 3 masters and 3 slaves. Again it’s not needed unless you start moving these nodes to additional EC2 instances or your application usage grows to handle the larger demand. With this design you can always add additional nodes later without application coding changes.

So here are the steps to create the cluster:

  1. create a minimum 6 Redis node instances with different hosts or ports using the following changes to the Redis conf file. To do this I created a /redis/data directory and copied the initial install 6379.conf file to the new port name in the /etc/redis directory. Then change each with the following
    port 6101
    
    pidfile /redis/data/redis_pid_6101.pid
    logfile /redis/data/redis_log_6101.log
    
    dbfilename dump_6101.rdb
    appendfilename "appendonly_6101.aof"
    cluster-config-file nodes_6101.conf
    
    requirepass myWickedLong256CharacterHashPassword
    
    dir /redis/data
    
    protected-mode no
    appendonly yes
    cluster-enabled yes
    
    # USE CLUSTER SYS INSTALL DEFAULTS BELOW
    cluster-node-timeout 15000
    cluster-slave-validity-factor 10
    cluster-migration-barrier 1
    cluster-require-full-coverage yes
    
    # USE OTHER SYS INSTALL DEFAULTS

    * create a .conf file for each port 6101 – 6106

  2. start each node with the redis-server command
    /usr/local/bin/redis-server /etc/redis/6101.conf

    * start each port 6101 – 6106

  3. Now we need to hack the redis-trib.rb progam with the following changes:
    This code change starts around line 57 and goes to line 125. You can cut, copy, and past as long as you get the exact same section of code (using Redis version 3.2.6) or simply scan through my code for the lines added and changed that are tagged with # tlr <start/end> comments

    class ClusterNode
     def initialize(addr)
     s = addr.split(":")
     if s.length < 2  puts "Invalid IP or Port (given as #{addr}) - use IP:Port format"  exit 1  end # tlr start pwd = nil  if s.length == 3  pwd = s.pop  end # tlr end port = s.pop # removes port from split array  ip = s.join(":") # if s.length > 1 here, it's IPv6, so restore address
     @r = nil
     @info = {}
     @info[:host] = ip
     @info[:port] = port
     @info[:slots] = {}
     @info[:migrating] = {}
     @info[:importing] = {}
     @info[:replicate] = false
    
    # tlr start
    @info[:password] = pwd 
    # tlr end
    
    @dirty = false # True if we need to flush slots info into node.
     @friends = []
     end
    
    def friends
     @friends
     end
    
    def slots
     @info[:slots]
     end
    
    def has_flag?(flag)
     @info[:flags].index(flag)
     end
    
    def to_s
     "#{@info[:host]}:#{@info[:port]}"
     end
    
    def connect(o={})
     return if @r
     print "Connecting to node #{self}: " if $verbose
     STDOUT.flush
     begin
    
    # tlr start
     if @info[:password] != nil
     @r = Redis.new(:host => @info[:host], :port => @info[:port], :timeout => 60, :password=>@info[:password])
     @r.ping
     else
     @r = Redis.new(:host => @info[:host], :port => @info[:port], :timeout => 60)
     @r.ping
     end
    # tlr end (the 2 lines in the else section are not changed from original)
    
    rescue
     xputs "[ERR] Sorry, can't connect to node #{self}"
     exit 1 if o[:abort]
     @r = nil
     end
     xputs "OK" if $verbose
     end
  4. next run the redis-trib.rb program to combine your nodes into one cluster. This may be a super long command from the command line especially if you have 256 character passwords but it works. (do it all on one line)
    /redis/redis-3.2.6/src/redis-trib.rb create --replicas 1 
    127.0.0.1:6101:my256charPassword 127.0.0.1:6102:my256charPassword 
    127.0.0.1:6103:my256charPassword 127.0.0.1:6104:my256charPassword 
    127.0.0.1:6105:my256charPassword 127.0.0.1:6106:my256charPassword

    * I did notice this produced a few errors as shown below but they are simply the process verification errors and the nodes are working fine.

    >>> Creating cluster
    >>> Performing hash slots allocation on 6 nodes...
    Using 3 masters:
    127.0.0.1:6101
    127.0.0.1:6102
    127.0.0.1:6103
    Adding replica 127.0.0.1:6104 to 127.0.0.1:6101
    Adding replica 127.0.0.1:6105 to 127.0.0.1:6102
    Adding replica 127.0.0.1:6106 to 127.0.0.1:6103
    
    (slot master/slave identifiers)
    
    Can I set the above configuration? (type 'yes' to accept): yes
    >>> Nodes configuration updated
    >>> Assign a different config epoch to each node
    >>> Sending CLUSTER MEET messages to join the cluster
    Waiting for the cluster to join.....
    [ERR] Sorry, can't connect to node 127.0.0.1:6105
    [ERR] Sorry, can't connect to node 127.0.0.1:6106
    [ERR] Sorry, can't connect to node 127.0.0.1:6103
    [ERR] Sorry, can't connect to node 127.0.0.1:6102
    [ERR] Sorry, can't connect to node 127.0.0.1:6104
    >>> Performing Cluster Check (using node 127.0.0.1:6101)
    M: 4f531ed4bcfd058b688a8692138fbdcc01a9dc7e 127.0.0.1:6101
     slots:0-5460 (5461 slots) master
     0 additional replica(s)
    [OK] All nodes agree about slots configuration.
    >>> Check for open slots...
    >>> Check slots coverage...
    [ERR] Not all 16384 slots are covered by nodes.

    A few more edits would fix the warning errors. 🙂 Since this is a one time command to initially setup your cluster, it’s not an issue. To add nodes to your existing cluster in the future you will user the rediscli command line tool with the CLUSTER MEET command.

  5. Confirm the cluster is working with the rediscli command setting a value.
    /usr/local/bin/redis-cli -c -p 6101 -a my256CharPassword
    SET foo bar
    GET foo
    CLUSTER SLOTS
    CLUSTER INFO

    You might notice foo gets pushed to a slot on the 2nd master. try SET a a and then try SET z z. You can also connect to any of the six nodes (6101 – 6106) to verify the sets with a GET command. (GET foo)

That’s all there is to it. You can open the 6101 – 6106 port to your local laptop and start developing on your local machine using the node.js ioredis client package on NPM at https://www.npmjs.com/package/ioredis

ps. Of course that’s not all! 🙂 additional code changes would be needed for example the slave to master login with AUTH.

Redis Session and MySQL Login

The following does a Redis session store and MySQL user register and login as well as a simple message post. This is run on a node.js server

package.json

{
 "name": "users",
 "version": "1.0.0",
 "description": "Register User",
 "main": "app.js",
 "script": "./app.js",
 "watch": true,
 "ignore_watch": ["node_modules"],
 "keywords": [
"login"
 ],
 "author": "Todd Rodzen",
 "license": "MIT",
 "dependencies": {
 "async": "^1.2.1",
 "body-parser": "^1.13.0",
 "connect-redis": "^2.3.0",
 "cookie-parser": "^1.3.5",
 "ejs": "^2.3.1",
 "express": "^4.14.0",
 "express-session": "^1.11.3",
 "mysql": "^2.7.0",
 "redis": "^0.12.1"
 }
}

app.js

/**
 Loading all dependencies.
**/
var express = require("express");
var redis = require("redis");
var mysql = require("mysql");
var session = require('express-session');
var redisStore = require('connect-redis')(session);
var bodyParser = require('body-parser');
var cookieParser = require('cookie-parser');
var path = require("path");
var async = require("async");
var client = redis.createClient();
var app = express();
var router = express.Router();

// Always use MySQL pooling.
// Helpful for multiple connections.

var pool = mysql.createPool({
 connectionLimit : 100,
 host : 'hmmmmm',
 user : 'you',
 password : 'ssshhhhh',
 database : 'hmmmm',
 debug : false
});

app.set('views', 'view');
app.engine('html', require('ejs').renderFile);

// IMPORTANT
// Here we tell Express to use Redis as session store.
// We pass Redis credentials and port information.
// And express does the rest ! 

app.use(session({
 secret: 'topics-session',
 store: new redisStore({ host: 'localhost', port: 6379, client: client,ttl : 260}),
 saveUninitialized: false,
 resave: false
}));
app.use(cookieParser("secretSign#143_!223"));
app.use(bodyParser.urlencoded({extended: false}));
app.use(bodyParser.json());

// This is an important function.
// This function does the database handling task.
// We also use async here for control flow.

function handle_database(req,type,callback) {
 async.waterfall([
 function(callback) {
 pool.getConnection(function(err,connection){
 if(err) {
 // if there is error, stop right away.
 // This will stop the async code execution and goes to last function.
 callback(true);
 } else {
 callback(null,connection);
 }
 });
 },
 function(connection,callback) {
 var SQLquery;
 switch(type) {
 case "login" :
 SQLquery = "SELECT * from user_login WHERE user_email='"+req.body.user_email+"' AND `user_password`='"+req.body.user_password+"'";
 break;
 case "checkEmail" :
 SQLquery = "SELECT * from user_login WHERE user_email='"+req.body.user_email+"'";
 break;
 case "register" :
 SQLquery = "INSERT into user_login(user_email,user_password,user_name) VALUES ('"+req.body.user_email+"','"+req.body.user_password+"','"+req.body.user_name+"')";
 break;
 case "addStatus" :
 SQLquery = "INSERT into msg_text(user_id,msg_text) VALUES ("+req.session.key["user_id"]+",'"+req.body.status+"')";
 break;
 case "getStatus" :
 SQLquery = "SELECT * FROM msg_text WHERE user_id="+req.session.key["user_id"];
 break;
 default :
 break;
 }
 callback(null,connection,SQLquery);
 },
 function(connection,SQLquery,callback) {
 connection.query(SQLquery,function(err,rows){
 connection.release();
 if(!err) {
 if(type === "login") {
 callback(rows.length === 0 ? false : rows[0]);
 } else if(type === "getStatus") {
 callback(rows.length === 0 ? false : rows);
 } else if(type === "checkEmail") {
 callback(rows.length === 0 ? false : true);
 } else {
 callback(false);
 }
 } else {
 // if there is error, stop right away.
 // This will stop the async code execution and goes to last function.
 callback(true);
 }
 });
 }],
 function(result){
 // This function gets call after every async task finished.
 if(typeof(result) === "boolean" && result === true) {
 callback(null);
 } else {
 callback(result);
 }
 });
}

/**
 --- Router Code begins here.
**/

router.get('/',function(req,res){
 res.render('index.html');
});

router.post('/login',function(req,res){
 handle_database(req,"login",function(response){
 if(response === null) {
 res.json({"error" : "true","message" : "Database error occured"});
 } else {
 if(!response) {
 res.json({
 "error" : "true",
 "message" : "Login failed ! Please register"
 });
 } else {
 req.session.key = response;
 res.json({"error" : false,"message" : "Login success."});
 }
 }
 });
});

router.get('/home',function(req,res){
 if(req.session.key) {
 res.render("home.html",{ email : req.session.key["user_name"]});
 } else {
 res.redirect("/");
 }
});

router.get("/fetchStatus",function(req,res){
 if(req.session.key) {
 handle_database(req,"getStatus",function(response){
 if(!response) {
 res.json({"error" : false, "message" : "There is no status to show."});
 } else {
 res.json({"error" : false, "message" : response});
 }
 });
 } else {
 res.json({"error" : true, "message" : "Please login first."});
 }
});

router.post("/addStatus",function(req,res){
 if(req.session.key) {
 handle_database(req,"addStatus",function(response){
 if(!response) {
 res.json({"error" : false, "message" : "Status is added."});
 } else {
 res.json({"error" : false, "message" : "Error while adding Status"});
 }
 });
 } else {
 res.json({"error" : true, "message" : "Please login first."});
 }
});

router.post("/register",function(req,res){
 handle_database(req,"checkEmail",function(response){
 if(response === null) {
 res.json({"error" : true, "message" : "This email is already present"});
 } else {
 handle_database(req,"register",function(response){
 if(response === null) {
 res.json({"error" : true , "message" : "Error while adding user."});
 } else {
 req.session.key = response;
 res.json({"error" : false, "message" : "Registered successfully."});
 }
 });
 }
 });
});

router.get('/logout',function(req,res){
 if(req.session.key) {
 req.session.destroy(function(){
 res.redirect('/');
 });
 } else {
 res.redirect('/');
 }
});

app.use('/',router);

app.listen(4201,function(){
 console.log("I am running at 4201");
});

view/index.html (code)
https://github.com/trodzen/MySQL-Redis-Session-Register/blob/master/view/index.html

view/home.html (code)
https://github.com/trodzen/MySQL-Redis-Session-Register/blob/master/view/home.html

You will need a working Redis db structure and MySQL and the two files used on the select/update SQL statements.

Try it, it’s easy.
That’s All Folks!

Redis on Amazon Linux

The YUM installed Redis version on Amazon Linux is an older version so we will go through the steps to install Redis 3.2.6.

redis

  1. sudo -i
  2. yum update
  3. yum install -y gcc*
  4. yum install -y tcl
  5. mkdir /redis
  6. sudo chmod 2775 /redis
  7. cd /redis
  8. wget http://download.redis.io/releases/redis-3.2.6.tar.gz
  9. tar xzf redis-3.2.6.tar.gz
  10. cd redis-3.2.6
  11. make
  12. make test
  13. make install
  14. cd utils
  15. chmod +x install_server.sh
  16. ./install_server.sh
    install with the following values:

    Welcome to the redis service installer
    This script will help you easily set up a running redis server
    
    Please select the redis port for this instance: [6379]
    Selecting default: 6379
    Please select the redis config file name [/etc/redis/6379.conf]
    Selected default - /etc/redis/6379.conf
    Please select the redis log file name [/var/log/redis_6379.log]
    Selected default - /var/log/redis_6379.log
    Please select the data directory for this instance [/var/lib/redis/6379]
    Selected default - /var/lib/redis/6379
    Please select the redis executable path [] /usr/local/bin/redis-server
    Selected config:
    Port : 6379
    Config file : /etc/redis/6379.conf
    Log file : /var/log/redis_6379.log
    Data dir : /var/lib/redis/6379
    Executable : /usr/local/bin/redis-server
    Cli Executable : /usr/local/bin/redis-cli
  17. chkconfig –level 2345 redis_6379 on
  18. chmod 2775 /etc/redis
  19. chmod 664 /etc/redis/6379.conf
  20. edit the /etc/redis/6379.conf file to set a password

That’s All!

ps. Want to set a password on the Redis store? It’s not recommended because a password doesn’t do much to secure a fast memory based key storage when 1000’s of password auth attempts can be thrown at it PER SECOND!  But on the other hand, if you want to add a password to prevent simply prying eyes like inhouse staff that won’t go through the trouble of building a password hacking program. Maybe you think the added password might prevent an inadvertent command to you Redis database like an accidental delete. Regardless the reason, here’s what you need to do:

edit the /etc/redis/6379.conf file
1. add your password to the password line and uncomment it. make it realy long.
2. turn protect mode off in the same file be commenting that line out.
3. (optional) comment the bind statement to allow connections from any interface. If you do this, you better control that port somewhere else, maybe with an AWS security group.

edit the  /etc/init.d/redis_6379 file and add the following command in the start and stop case procedures:

echo "Using Auth Password"
CLIEXEC="/usr/local/bin/redis-cli -a mywickedLong256character?Password"

Now you can do a sudo service redis_6379 restart command.

Powered by WordPress.com.

Up ↑