Skip to content

Commit

Permalink
Add MD5 hashing algorithm, add copyright and license notice to readme…
Browse files Browse the repository at this point in the history
…; modify postsMetadata storage to use key entry in redis and store under object hash; update /stats-data-json to retrieve from newly formated postsMetadata storage
  • Loading branch information
thrize committed Jan 22, 2017
1 parent 89dfc1d commit 5c59b24
Show file tree
Hide file tree
Showing 4 changed files with 291 additions and 12 deletions.
10 changes: 7 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,9 +47,15 @@ Make sure to read the instructions though! Heroku has a basic free plan but if n

All original programming is under the CC0 license and so it completely open and free to use in any capacity. It's in the spirit of the project that it is open to all.

The [steem Node.js package](https://www.npmjs.com/package/steem) by adcpm is central to the app, a big thank you to the creator. Please [star it on GitHub](https://github.com/adcpm/steem) to support their development.
Included in this repo are the following libraries:

- [Bootstrap](https://getbootstrap.com/) is used for web frontend, and so included in this repo, and is under the MIT license, copyright to Twitter
- MD5 Hashing algorithm ([modified from this source](http://www.queness.com/code-snippet/6523/generate-md5-hash-with-javascript) by Paul Johnston and Greg Holt is under copyright and licensed under the BSD license. Legal text [available here](http://pajhome.org.uk/site/legal.html)

The [steem Node.js package](https://www.npmjs.com/package/steem) by adcpm is central to the app, a big thank you to the creators. Please [star it on GitHub](https://github.com/adcpm/steem) to support their development and check out their project [Busy](https://github.com/adcpm/busy).

Several other Node NPM libraries are used as dependencies (thier source is not included in this repo). Thanks to their creators!

- [express](https://www.npmjs.com/package/express) and [body-parser](https://www.npmjs.com/package/body-parser) by dougwilson, as basic glue used by nearly every Node.js app
- [sendgrid](https://www.npmjs.com/package/sendgrid) by thinkingserious, to send email notifications
- [Q](https://www.npmjs.com/package/q) by kriskowal, to promise-ify and de-callback-hell-ify the long process of running a bot iteration
Expand All @@ -60,8 +66,6 @@ Several other Node NPM libraries are used as dependencies (thier source is not i
- [retext](https://www.npmjs.com/package/retext) and [retext-sentiment](https://www.npmjs.com/package/retext-sentiment) also by wooorm, for determining sentiment using NLP
- [wait.for](https://www.npmjs.com/package/wait.for) by luciotato, for turning async functions into sync functions

Additionally [Bootstrap](https://getbootstrap.com/) is used for web frontend, and so included in this repo, and is under the MIT license copyright to Twitter.

## Disclaimer

We are not required to supply terms because we are not running a service. However obviously you are at your own liability if you use this software.
Expand Down
186 changes: 186 additions & 0 deletions extra.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,186 @@
/*
* The MD5 Hashing algorithm from http://www.queness.com/code-snippet/6523/generate-md5-hash-with-javascript
*
* The only modifications were to the superficial formatting, and adding explicit "var" declarations for new variables, as is required for
* WebExtension JavaScript style.
*
* The license as available at http://pajhome.org.uk/site/legal.html is copied here in full:
*
* The JavaScript code implementing the algorithm is derived from the C code in RFC 1321 and is covered by the following copyright:
* License to copy and use this software is granted provided that it is identified as the "RSA Data Security, Inc. MD5 Message-Digest Algorithm" in all material mentioning or referencing this software or this function.
* License is also granted to make and use derivative works provided that such works are identified as "derived from the RSA Data Security, Inc. MD5 Message-Digest Algorithm" in all material mentioning or referencing the derived work.
* RSA Data Security, Inc. makes no representations concerning either the merchantability of this software or the suitability of this software for any particular purpose. It is provided "as is" without express or implied warranty of any kind.
* These notices must be retained in any copies of any part of this documentation and/or software.
* This copyright does not prohibit distribution of the JavaScript MD5 code under the BSD license.
*/

/*
* A JavaScript implementation of the RSA Data Security, Inc. MD5 Message
* Digest Algorithm, as defined in RFC 1321.
* Copyright (C) Paul Johnston 1999 - 2000.
* Updated by Greg Holt 2000 - 2001.
* See http://pajhome.org.uk/site/legal.html for details.
*/

/*
* Convert a 32-bit number to a hex string with ls-byte first
*/
var hex_chr = "0123456789abcdef";
function rhex(num) {
var str = "";
for(var j = 0 ; j <= 3 ; j++) {
str += hex_chr.charAt((num >> (j * 8 + 4)) & 0x0F) +
hex_chr.charAt((num >> (j * 8)) & 0x0F);
}
return str;
}

/*
* Convert a string to a sequence of 16-word blocks, stored as an array.
* Append padding bits and the length, as described in the MD5 standard.
*/
function str2blks_MD5(str) {
var nblk = ((str.length + 8) >> 6) + 1;
var blks = new Array(nblk * 16);
for (var i = 0 ; i < nblk * 16 ; i++) {
blks[i] = 0;
}
for (var i = 0 ; i < str.length ; i++) {
blks[i >> 2] |= str.charCodeAt(i) << ((i % 4) * 8);
}
blks[i >> 2] |= 0x80 << ((i % 4) * 8);
blks[nblk * 16 - 2] = str.length * 8;
return blks;
}

/*
* Add integers, wrapping at 2^32. This uses 16-bit operations internally
* to work around bugs in some JS interpreters.
*/
function add(x, y) {
var lsw = (x & 0xFFFF) + (y & 0xFFFF);
var msw = (x >> 16) + (y >> 16) + (lsw >> 16);
return (msw << 16) | (lsw & 0xFFFF);
}

/*
* Bitwise rotate a 32-bit number to the left
*/
function rol(num, cnt) {
return (num << cnt) | (num >>> (32 - cnt));
}

/*
* These functions implement the basic operation for each round of the
* algorithm.
*/
function cmn(q, a, b, x, s, t) {
return add(rol(add(add(a, q), add(x, t)), s), b);
}
function ff(a, b, c, d, x, s, t) {
return cmn((b & c) | ((~b) & d), a, b, x, s, t);
}
function gg(a, b, c, d, x, s, t) {
return cmn((b & d) | (c & (~d)), a, b, x, s, t);
}
function hh(a, b, c, d, x, s, t) {
return cmn(b ^ c ^ d, a, b, x, s, t);
}
function ii(a, b, c, d, x, s, t) {
return cmn(c ^ (b | (~d)), a, b, x, s, t);
}

/*
* Take a string and return the hex representation of its MD5.
*/
function calcMD5(str) {
var x = str2blks_MD5(str);
var a = 1732584193;
var b = -271733879;
var c = -1732584194;
var d = 271733878;

for (var i = 0; i < x.length; i += 16) {
var olda = a;
var oldb = b;
var oldc = c;
var oldd = d;

a = ff(a, b, c, d, x[i+ 0], 7 , -680876936);
d = ff(d, a, b, c, x[i+ 1], 12, -389564586);
c = ff(c, d, a, b, x[i+ 2], 17, 606105819);
b = ff(b, c, d, a, x[i+ 3], 22, -1044525330);
a = ff(a, b, c, d, x[i+ 4], 7 , -176418897);
d = ff(d, a, b, c, x[i+ 5], 12, 1200080426);
c = ff(c, d, a, b, x[i+ 6], 17, -1473231341);
b = ff(b, c, d, a, x[i+ 7], 22, -45705983);
a = ff(a, b, c, d, x[i+ 8], 7 , 1770035416);
d = ff(d, a, b, c, x[i+ 9], 12, -1958414417);
c = ff(c, d, a, b, x[i+10], 17, -42063);
b = ff(b, c, d, a, x[i+11], 22, -1990404162);
a = ff(a, b, c, d, x[i+12], 7 , 1804603682);
d = ff(d, a, b, c, x[i+13], 12, -40341101);
c = ff(c, d, a, b, x[i+14], 17, -1502002290);
b = ff(b, c, d, a, x[i+15], 22, 1236535329);

a = gg(a, b, c, d, x[i+ 1], 5 , -165796510);
d = gg(d, a, b, c, x[i+ 6], 9 , -1069501632);
c = gg(c, d, a, b, x[i+11], 14, 643717713);
b = gg(b, c, d, a, x[i+ 0], 20, -373897302);
a = gg(a, b, c, d, x[i+ 5], 5 , -701558691);
d = gg(d, a, b, c, x[i+10], 9 , 38016083);
c = gg(c, d, a, b, x[i+15], 14, -660478335);
b = gg(b, c, d, a, x[i+ 4], 20, -405537848);
a = gg(a, b, c, d, x[i+ 9], 5 , 568446438);
d = gg(d, a, b, c, x[i+14], 9 , -1019803690);
c = gg(c, d, a, b, x[i+ 3], 14, -187363961);
b = gg(b, c, d, a, x[i+ 8], 20, 1163531501);
a = gg(a, b, c, d, x[i+13], 5 , -1444681467);
d = gg(d, a, b, c, x[i+ 2], 9 , -51403784);
c = gg(c, d, a, b, x[i+ 7], 14, 1735328473);
b = gg(b, c, d, a, x[i+12], 20, -1926607734);

a = hh(a, b, c, d, x[i+ 5], 4 , -378558);
d = hh(d, a, b, c, x[i+ 8], 11, -2022574463);
c = hh(c, d, a, b, x[i+11], 16, 1839030562);
b = hh(b, c, d, a, x[i+14], 23, -35309556);
a = hh(a, b, c, d, x[i+ 1], 4 , -1530992060);
d = hh(d, a, b, c, x[i+ 4], 11, 1272893353);
c = hh(c, d, a, b, x[i+ 7], 16, -155497632);
b = hh(b, c, d, a, x[i+10], 23, -1094730640);
a = hh(a, b, c, d, x[i+13], 4 , 681279174);
d = hh(d, a, b, c, x[i+ 0], 11, -358537222);
c = hh(c, d, a, b, x[i+ 3], 16, -722521979);
b = hh(b, c, d, a, x[i+ 6], 23, 76029189);
a = hh(a, b, c, d, x[i+ 9], 4 , -640364487);
d = hh(d, a, b, c, x[i+12], 11, -421815835);
c = hh(c, d, a, b, x[i+15], 16, 530742520);
b = hh(b, c, d, a, x[i+ 2], 23, -995338651);

a = ii(a, b, c, d, x[i+ 0], 6 , -198630844);
d = ii(d, a, b, c, x[i+ 7], 10, 1126891415);
c = ii(c, d, a, b, x[i+14], 15, -1416354905);
b = ii(b, c, d, a, x[i+ 5], 21, -57434055);
a = ii(a, b, c, d, x[i+12], 6 , 1700485571);
d = ii(d, a, b, c, x[i+ 3], 10, -1894986606);
c = ii(c, d, a, b, x[i+10], 15, -1051523);
b = ii(b, c, d, a, x[i+ 1], 21, -2054922799);
a = ii(a, b, c, d, x[i+ 8], 6 , 1873313359);
d = ii(d, a, b, c, x[i+15], 10, -30611744);
c = ii(c, d, a, b, x[i+ 6], 15, -1560198380);
b = ii(b, c, d, a, x[i+13], 21, 1309151649);
a = ii(a, b, c, d, x[i+ 4], 6 , -145523070);
d = ii(d, a, b, c, x[i+11], 10, -1120210379);
c = ii(c, d, a, b, x[i+ 2], 15, 718787259);
b = ii(b, c, d, a, x[i+ 9], 21, -343485551);

a = add(a, olda);
b = add(b, oldb);
c = add(c, oldc);
d = add(d, oldd);
}
return rhex(a) + rhex(b) + rhex(c) + rhex(d);
}

/* Set public API */
module.exports.calcMD5 = calcMD5;
81 changes: 73 additions & 8 deletions lib.js
Original file line number Diff line number Diff line change
Expand Up @@ -73,14 +73,18 @@ const
stripMarkdownProcessor = remark().use(strip),
retext = require('retext'),
sentiment = require('retext-sentiment'),
wait = require('wait.for');
wait = require('wait.for'),
extra = require('/extra.js');

const
MINNOW = 0,
DOLPHIN = 1,
WHALE = 2;

const
MILLIS_IN_DAY = 86400000;

var
MAX_POST_TO_READ = 100,
CAPITAL_DOLPHIN_MIN = 25000,
CAPITAL_WHALE_MIN = 100000,
Expand All @@ -89,7 +93,8 @@ const
SCORE_THRESHOLD_INC_PC = 0.5,
NUM_POSTS_FOR_AVG_WINDOW = 20,
MAX_VOTES_IN_24_HOURS = 40,
MIN_WORDS_FOR_ARTICLE = 100;
MIN_WORDS_FOR_ARTICLE = 100,
DAYS_KEEP_LOGS = 5;

/* Private variables */
var fatalError = false;
Expand Down Expand Up @@ -870,11 +875,11 @@ function runBot(callback, options) {
}
// and save postsMetadata to persistent
persistentLog(" - saving posts_metadata");
persistJson("posts_metadata", {posts_metadata: postsMetadata}, function(err) {
persistentLog(" - - ERROR SAVING posts_metadata");
});
// finish
deferred.resolve(true);
savePostsMetadata({postsMetadata: postsMetadata}, function(res) {
persistentLog(" - - SAVING posts_metadata: "+res.message);
// finish
deferred.resolve(true);
})
return deferred.promise;
},
// cast votes to steem
Expand Down Expand Up @@ -1329,6 +1334,65 @@ function updateMetricList(list, contents, apiKey, callback) {
});
}

function savePostsMetadata(postsMetadataObj, callback) {
console.log("savePostsMetadata");
var keys = null;
try {
keys = wait.for(redisClient.get, "postsMetadata_keys");
} catch(err) {
console.log(" - postsMetadata_keys doesn't exist, probably first time run, will create newly");
}
try {
var toKeep = [];
if (keys != null) {
var keysObj = JSON.parse(keys);
console.log(" - removing old keys");
// mark old keys for deletion, to clear space before saving
var toDelete = [];
for (var i = 0 ; i < keysObj.keys.length ; i++) {
if (((new Date()).getTime() - keysObj.keys[i].date) > (DAYS_KEEP_LOGS * MILLIS_IN_DAY)) {
toDelete.push(keysObj.keys[i].key);
} else {
toKeep.push(keysObj.keys[i]);
}
}
console.log(" - - keeping "+toKeep.length+" keys");
console.log(" - - deleting "+toDelete.length+" keys");
for (var i = 0 ; i < toDelete.length ; i++) {
var result = wait.for(redisClient.del, keysObj.keys[i]);
if (result > 0) {
console.log(" - - - deleted redis key: "+key)
} else {
console.log(" - - - COULDNT delete redis key: "+key)
}
}
}
var stringifiedJson = JSON.stringify(postsMetadataObj);
var key = extra.calcMD5(stringifiedJson);
console.log(" - adding new postsMetadata key: "+key);
toKeep.push({date: (new Date()).getTime(), key: key});
wait.for(redisClient.set, "postsMetadata_keys", JSON.stringify({keys: toKeep});
console.log(" - adding new postsMetadata under key: "+key);
wait.for(redisClient.set, key, stringifiedJson);
console.log(" - finished saving postsMetadata");
callback({status: 200, message: "savePostsMetadata, success, saved postsMetadata with key: "+key});
} catch(err) {
console.log("savePostsMetadata, error: "+err.message);
callback({status: 500, message: "savePostsMetadata, error: "+err.message});
}
}

function getPostsMetadataKeys(callback) {
try {
var keys = wait.for(redisClient.get, "postsMetadata_keys");
var keysObj = JSON.parse(keys);
callback(null, keysObj.keys);
} catch(err) {
console.log("getPostsMetadataKeys, error: "+err.message);
callback({status: 500, message: "getPostsMetadataKeys, error: "+err.message}, []);
}
}

/*
* Steem Utils
*/
Expand Down Expand Up @@ -1495,4 +1559,5 @@ module.exports.getPersistentJson = getPersistentJson;
module.exports.getPersistentString = getPersistentString;
module.exports.updateWeightMetric = updateWeightMetric;
module.exports.deleteWeightMetric = deleteWeightMetric;
module.exports.updateMetricList = updateMetricList;
module.exports.updateMetricList = updateMetricList;
module.exports.getPostsMetadataKeys = getPostsMetadataKeys;
26 changes: 25 additions & 1 deletion server.js
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,10 @@ const
express = require("express"),
path = require("path"),
bodyParser = require("body-parser"),
fs = require('fs');
fs = require('fs'),
redis = require("redis"),
redisClient = require('redis').createClient(process.env.REDIS_URL),
wait = require('wait.for');

var html_algo_emptyList = "<tr><td>None</td><td></td><td>-</td><td>-</td><th><p><a class=\"btn btn-default\" href=\"#\" role=\"button\"><strike>Delete<strike></a></p></th></tr>";
var html_test_emptyList = "<tr><td>None</td><td>-</td>-<td></tr>";
Expand Down Expand Up @@ -209,6 +212,25 @@ app.get("/stats-data-json", function(req, res) {
handleError(res, "/stats-data-json Unauthorized", "stats-data-json: api_key invalid", 401);
return;
}
lib.getPostsMetadataKeys(function(err, keys) {
if (err) {
handleErrorJson(res, "/stats-data-json Server error", "stats-data-json: no data in store, no keys", 500);
return;
}
console.log(" - /stats-data-json got keys: "+JSON.stringify(keys));
try {
var postsMetadataList = [];
for (var i = 0 ; i < keys.length ; i++) {
var postsMetadataObj = wait.for(redisClient.get, keys[i]);
postsMetadataList.push(postsMetadataObj);
}
res.json({postsMetadataList: postsMetadataList});
} catch(err) {
handleErrorJson(res, "/stats-data-json Server error", "stats-data-json: error fetching data: "+err.message, 500);
return;
}
});
/*
lib.getPersistentJson("posts_metadata", function(postsMetadata) {
console.log("attempted to get postsMetadata: "+postsMetadata);
if (postsMetadata != null) {
Expand All @@ -217,8 +239,10 @@ app.get("/stats-data-json", function(req, res) {
handleErrorJson(res, "/stats-data-json Unauthorized", "stats-data-json: no data in store", 500);
}
});
*/
});


app.get("/get-algo", function(req, res) {
if (!req.query.api_key) {
handleError(res, "/get-algo Unauthorized", "get-algo: api_key not supplied", 401);
Expand Down

0 comments on commit 5c59b24

Please sign in to comment.