Skip to content
This repository was archived by the owner on Sep 19, 2018. It is now read-only.

Commit

Permalink
initial commit
Browse files Browse the repository at this point in the history
  • Loading branch information
Sam Garrett committed Feb 3, 2016
0 parents commit 9633cea
Show file tree
Hide file tree
Showing 172 changed files with 13,382 additions and 0 deletions.
12 changes: 12 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
*.iml
.idea
ui/node_modules
ui/coverage
ui/deploy
ui/*.log
*.swp
chronos-web/src/main/resources/site
ui/public
!ui/public/ico
!ui/public/index.html
.env
27 changes: 27 additions & 0 deletions LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
Copyright (c) 2016 AOL Inc.
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.

3. Neither the name of the copyright holder nor the names of its contributors
may be used to endorse or promote products derived from this software without
specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
119 changes: 119 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
# chronos

> cron-like jobs for back-end systems
![UI](/../screenshots/ui.png?raw=true "UI")

## Features of _chronos_

* job history tracking
* job versions
* automatic rerun with configurable max attempts
* email notifications (reports, failures, etc)
* REST [api][api]
* multiple datasource support
* BSD [license][license]

[api]: chronos-web/src/main/java/com/huffingtonpost/chronos/servlet/ChronosController.java
[license]: LICENSE


## What is _chronos_?

_chronos_ is for running jobs that would normally be put into a crontab but
would be easier to manage and track through a simple ui and some retry and
emailing mechanisms.


### Example use-cases

A _chronos_ job could be a Hive Query that needs to run every hour to
populate a table that your api queries for an internal app, or just a periodic
query that transforms some data for another back-end process. Another example
might be a recurring report that needs to be sent to business leads every week,
that queries a MySql database and sends the top 100 results. _chronos_ makes
these tasks simple to create, update, track, and rerun.


## Why _chronos_?

You're probably tired of ssh-ing to machines and rerunning jobs manually after
hearing that they failed from someone that's not even on your tech team. So
we've built a system that takes care of that and let's you rest easy on the
weekend and hopefully spend more time coding during work hours.

_chronos_ has been implemented in a way that lets you add a new back-end
for storing jobs and history if needed. Currently we have a MySql
implementation of the [WithBackend interface][backend]. It would be trivial to
add more to suit your needs. JDBC connections are currently used for
jobs.

[backend]: chronos-agent/src/main/java/com/huffingtonpost/chronos/persist/WithBackend.java

## Configuration and Prerequisites

_chronos_ uses a Java-based [spring config][sc].

* java 7 for backend
* node >= 4 and npm >= 3 for ui

[sc]: chronos-web/src/main/java/com/huffingtonpost/chronos/servlet/TestConfig.java

## Test

mvn clean test


### Run locally

./build_site.sh
mvn clean package install
cd chronos-web/
mvn tomcat7:run

Note: This creates a small test table in H2 for running jobs against.
Details of the table can be found [here][table].

[table]: chronos-agent/src/test/java/com/huffingtonpost/chronos/agent/H2TestJobDaoImpl.java


#### Run a job with the ui

1. Navigate to http://localhost:8080
2. Click New Job
3. Fill out the form. A query similar to `CREATE TABLE my_table_name AS SELECT * FROM testchronos;`
will work.
4. View log output of `mvn tomcat7:run` to see that job has completed.


## Future development

[Pull-requests][pr] are welcome! Feel free to contact us with Github Issues.

[pr]: https://help.github.com/articles/using-pull-requests/


## Further Reading

[Reliable Cron Across the Planet](https://queue.acm.org/detail.cfm?id=2745840)

[Chronos: A Replacement for Cron](http://nerds.airbnb.com/introducing-chronos/)

[Quora's Distributed Cron Architecture](https://engineering.quora.com/Quoras-Distributed-Cron-Architecture)

[Distributed Job Scheduling for AWS](https://medium.com/aws-activate-startup-blog/distributed-job-scheduling-for-aws-1c9f984b336d#.ymfpq4jt9)

[Luigi](https://github.com/spotify/luigi)


## Authors

_chronos_ was built at [The Huffington Post](http://www.huffingtonpost.com/),
mainly by [@sinemetu1][sm1], [@mikestopcontinues][msc], [@edwardcapriolo][ec],
[@ellnuh][se], and [@dmitry-s][ds].

[sm1]: https://github.com/sinemetu1
[msc]: https://github.com/mikestopcontinues
[ec]: https://github.com/edwardcapriolo
[se]: https://www.instagram.com/ellnuh/
[ds]: https://github.com/dmitry-s
18 changes: 18 additions & 0 deletions build_site.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
#!/bin/bash

# build the ui

cd ui/
npm install
npm run deploy
cd ../

# move ui files in

siteDir=chronos-web/src/main/resources/site/
if [ -d "$siteDir" ] ; then
rm -rf "$siteDir/*"
else
mkdir "$siteDir"
fi
cp -r ui/public/* "$siteDir"
4 changes: 4 additions & 0 deletions chronos-agent/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
.classpath
.project
.settings/
target/
173 changes: 173 additions & 0 deletions chronos-agent/pom.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,173 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<parent>
<groupId>com.huffingtonpost</groupId>
<artifactId>chronos-parent</artifactId>
<version>1.0.0</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>chronos-agent</artifactId>
<packaging>jar</packaging>
<repositories>
<repository>
<id>cloudera-repo-releases</id>
<url>https://repository.cloudera.com/artifactory/repo/</url>
</repository>
</repositories>
<properties>
<powermock.version>1.6.1</powermock.version>
<jackson.version>2.6.5</jackson.version>
</properties>
<dependencies>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.1.37</version>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-all</artifactId>
<version>1.10.8</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.powermock</groupId>
<artifactId>powermock-module-junit4</artifactId>
<version>${powermock.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.powermock</groupId>
<artifactId>powermock-api-mockito</artifactId>
<version>${powermock.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.3.0</version>
<exclusions>
<exclusion>
<groupId>org.mortbay.jetty</groupId>
<artifactId>servlet-api</artifactId>
</exclusion>
<exclusion>
<groupId>javax.servlet</groupId>
<artifactId>servlet-api</artifactId>
</exclusion>
<exclusion>
<groupId>javax.servlet.jsp</groupId>
<artifactId>jsp-api</artifactId>
</exclusion>
<exclusion>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-core-asl</artifactId>
</exclusion>
<exclusion>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-mapper-asl</artifactId>
</exclusion>
<exclusion>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-jaxrs</artifactId>
</exclusion>
<exclusion>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-xc</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-service</artifactId>
<version>0.13.1-cdh5.3.0</version>
<exclusions>
<exclusion>
<groupId>org.mortbay.jetty</groupId>
<artifactId>servlet-api</artifactId>
</exclusion>
<exclusion>
<groupId>javax.servlet</groupId>
<artifactId>servlet-api</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-jdbc</artifactId>
<version>0.13.1-cdh5.3.0</version>
<exclusions>
<exclusion>
<groupId>org.mortbay.jetty</groupId>
<artifactId>servlet-api</artifactId>
</exclusion>
<exclusion>
<groupId>javax.servlet</groupId>
<artifactId>servlet-api</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.h2database</groupId>
<artifactId>h2</artifactId>
<version>1.4.190</version>
</dependency>
<dependency>
<groupId>joda-time</groupId>
<artifactId>joda-time</artifactId>
<version>2.6</version>
</dependency>
<dependency>
<groupId>io.teknek</groupId>
<artifactId>graphite-reporter</artifactId>
<version>0.0.1</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
<version>${jackson.version}</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
<version>${jackson.version}</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>${jackson.version}</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.datatype</groupId>
<artifactId>jackson-datatype-joda</artifactId>
<version>${jackson.version}</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.8.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.15</version>
<type>jar</type>
<scope>compile</scope>
<exclusions>
<exclusion>
<groupId>javax.jms</groupId>
<artifactId>jms</artifactId>
</exclusion>
<exclusion>
<groupId>com.sun.jdmk</groupId>
<artifactId>jmxtools</artifactId>
</exclusion>
<exclusion>
<groupId>com.sun.jmx</groupId>
<artifactId>jmxri</artifactId>
</exclusion>
</exclusions>
</dependency>
</dependencies>
</project>
Loading

0 comments on commit 9633cea

Please sign in to comment.