Skip to content

Navigation Menu

Explore
By company size
By use case
By industry
View all solutions
Topics
- AI
- DevOps
- Security
- Software Development
- View all
Explore
- GitHub Sponsors
  Fund open source developers
- The ReadME Project
  GitHub community articles
Repositories
- Enterprise platform
  AI-powered developer platform
Available add-ons
Pricing

Search code, repositories, users, issues, pull requests...

Search

Clear

Search syntax tips

Provide feedback

We read every piece of feedback, and take your input very seriously.

Include my email address so I can be contacted

Saved searches

Use saved searches to filter your results more quickly

Name

Query

To see all available qualifiers, see our documentation.

You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session.

Dismiss alert

PaulMcInnis / JobFunnel Public

Notifications You must be signed in to change notification settings
Fork 227
Star 1.9k

Code
Issues 8
Pull requests
Discussions
Actions
Projects
Wiki
Security
Insights

Additional navigation options

Code
Issues
Pull requests
Discussions
Actions
Projects
Wiki
Security
Insights

JobFunnel 3.0 with localization, ABC and improved scraping#90

Merged

PaulMcInnis merged 66 commits intomasterPaulMcInnis/JobFunnel:masterfrom ABCJobFunnelPaulMcInnis/JobFunnel:ABCJobFunnelCopy head branch name to clipboard

Sep 12, 2020

+4,315-4,548

Conversation Commits66 (66)Checks Files changed

Commits

Commits on Sep 10, 2020

initial work
PaulMcInnis
committed
got scraping for indeed going again, with legacy CSV support
PaulMcInnis
committed
got delaying with DelayConfig and TFIDF + other filters working again
PaulMcInnis
committed
fix some status parsing issues
PaulMcInnis
committed
connected CLI parser, making no changes to args
PaulMcInnis
committed
getting glassdoor going again
PaulMcInnis
committed
more progress, converging on having job setters within base scraper, and using a locale in YAML
PaulMcInnis
committed
more progress, working on the CLI side now. looking to streamline things here with some basic validation
PaulMcInnis
committed
add cerberus schema validator
PaulMcInnis
committed
implemented config and validation with Cerberus
PaulMcInnis
committed
Got indeed going once more, now with the abstract job creation
PaulMcInnis
committed
moved bs4 parser into JobFunnelConfig for future CLI/cfg impl
PaulMcInnis
committed
Got set() and get() working with lots of assertions to keep it safe. Also added a progress bar and cleaned up the logs a bit
PaulMcInnis
committed
cleaning up some descriptions
PaulMcInnis
committed
got delaying back, but only for fields that need it
PaulMcInnis
committed
added --recover CLI and fixed stdout logging handler
PaulMcInnis
committed
Added back Monster scraper
PaulMcInnis
committed
Got GlassDoorStatic going again.
PaulMcInnis
committed
added Remote and wage jobfields and added wage scraping to GlassDoor static scraper
PaulMcInnis
committed
further improve defaults, schema and CLI interaction
PaulMcInnis
committed
fix demo settings YAML and make loggers display scraper name / jobfunnel
PaulMcInnis
committed
Added json entry property to Job, improved logging format, improved cache handling, moved job default init kwargs into BaseScraper property, made a re-usable get_logging method, reworking vectorize…
PaulMcInnis
committed
Improve jobfunnel behaviour in --no-scrape mode to prevent writing an empty CSV
PaulMcInnis
committed
Fixed monster scraping (multi-page), Fixed TFIDF filter and duplicates handling
PaulMcInnis
committed
Move is_old into Job, minimize JSON and CSV loading by setting as self and reorganizing the open()s
PaulMcInnis
committed
Make filtering dynamic so we can prevent scraping un-promising jobs as quickly as possible
PaulMcInnis
committed
moved update if newer into Job, Vastly improved duplicates detection workflow with DuplicateJob, Made raw scrape first set, added tag scraping to monster, Made JobFilter class
PaulMcInnis
committed
Make it so that we always scrape duplicates to ensure that we can update the existing jobs
PaulMcInnis
committed
Update readme.md

Paul McInnis
authored and
PaulMcInnis
committed
Formalize minimum required job fields a bit more, improve crash-out behavior for get/set, stop checking against duplicates dict when inspecting existing CSV.
PaulMcInnis
committed
Implement high-priority JobFields so we can stage set() operations + improve default retry behaviour
PaulMcInnis
committed
undo delay bypass
PaulMcInnis
committed
fix CLI defaults when mixing YAML and arguments and possibly defaults.
PaulMcInnis
committed
make demo not create hidden cache folder
PaulMcInnis
committed
Update readme.md
PaulMcInnis
committed
Fix flake8 issues without introducing circular imports + fix remaining issue with CLI + fix build test cli.
PaulMcInnis
committed
JobFunnelConfig -> JobFunnelConfigManager
PaulMcInnis
committed
Use get since config sub-keys can be missing.
PaulMcInnis
committed
Make logging handled through a Logger class which is inheritable
PaulMcInnis
committed
Add graphviz generator for call graphs + fix some minor leftovers
PaulMcInnis
committed
Fix delay synchronization between worker threads by controlling access with multiprocessing.Manager.Lock
PaulMcInnis
committed
Cleanup + calculate job get/set actions once in BaseScraper
PaulMcInnis
committed
Fixed some minor validation issues, moved path creation logic out of config manager
PaulMcInnis
committed
Fix block list file naming + remove create_dirs call
PaulMcInnis
committed
Fix PyEnv
PaulMcInnis
committed
Sig. improved CLI vs YAML vs defaults handling, upped python requires to 3.8, made duplicates list file mandatory.
PaulMcInnis
committed
min_delay -> min_duration
PaulMcInnis
committed
Swap over to pip from pipenv to fix build
PaulMcInnis
committed
fix comparison
PaulMcInnis
committed
Clean up imports + expand travis smoke testing to include USA
PaulMcInnis
committed
Fix numerous issues still existing in CLI.
PaulMcInnis
committed
remove import pdb!
PaulMcInnis
committed
Fix test module structure
PaulMcInnis
committed
fix CLI by restricting sub-commands to either load (YAML) or custom (CLI), updated readme to reflect this and also updated demo settings desc.
PaulMcInnis
committed
Remove .idea (too-user-specific), and Pipfile (PipEnv is no longer needed)
PaulMcInnis
committed
make log-level a base argument
PaulMcInnis
committed
Add some cli parser testing + fix more minor bugs in CLI parser
PaulMcInnis
committed
missed files
PaulMcInnis
committed
Fix directory creation for cache folder
PaulMcInnis
committed
Update travis to use new command format
PaulMcInnis
committed
simplify readme.md and remove some old demos I dont want to maintain
PaulMcInnis
committed
Add versioning to cache files, cleanup logo files, fix block list default and travis build
PaulMcInnis
committed

Commits on Sep 11, 2020

Update demo CSV image
PaulMcInnis
committed
Use tad preview b/c it's prettier + lower codecov standards for now
PaulMcInnis
committed
Refactor custom --> inline, resolve more pylint warnings, lower project codecov, clean up more imports, add inter-scraper-validity check, expand config validators
PaulMcInnis
committed
Fix domain validation + minor readme changes
PaulMcInnis
committed

Footer

© 2025 GitHub, Inc.

Footer navigation

Terms
Privacy
Security
Status
Docs
Contact

You can’t perform that action at this time.