NewsBlur is a personal news reader bringing people together to talk about the world.

Overview

NewsBlur

Get it on F-Droid Get it on Google Play

Features

  1. Shows the original site (you have to see it to believe it).
  2. Hides stories you don't want to read based on tags, keywords, authors, etc.
  3. Highlights stories you want to read, based on the same criteria.

Technologies

Server-side

  • Python 2.7+: The language of choice.
  • Django: Web framework written in Python, used to serve all pages.
  • Celery & RabbitMQ: Asynchronous queueing server, used to fetch and parse RSS feeds.
  • MongoDB, Pymongo, & Mongoengine: Non-relational database, used to store stories, read stories, feed/page fetch histories, and proxied sites.
  • PostgreSQL: Relational database, used to store feeds, subscriptions, and user accounts.
  • Redis: Programmer's database, used to assemble stories for the river, store story ids, manage feed fetching schedules, and the minuscule bit of caching that NewsBlur uses.
  • Elasticsearch: Search database, use for searching stories. Optional.

Client-side and design

  • jQuery: Cross-browser compliant JavaScript code. IE works without effort.
  • Underscore.js: Functional programming for JavaScript. Indispensable.
  • Backbone.js: Framework for the web app. Also indispensable.
  • Miscellaneous jQuery Plugins: Everything from resizable layouts, to progress bars, sortables, date handling, colors, corners, JSON, animations. See the complete list.

Installation Instructions

Prerequisites

What you can safely ignore

Not every program listed in the Prerequisites section is necessary to run NewsBlur.

  • Elasticsearch is the only module that requires Java. If you can live without searching for feeds or searching for stories, then you can ignore it and NewsBlur will just spit out that you don't have a search server in the logs.
  • Jammit is for asset compression. Don't bother using it since the alternative is to just serve every js and css file in individual files without compression. Besides, nginx gzips those files automatically if you use the built-in nginx config. Just set DEBUG_ASSETS = True in your local_settings.py (which is also in local_settings.py.template).
  • numpy and scipy are used for the colors used all over the site. Every site's favicon is analyzed for its dominant color, and that color is what gives every site its feel. You'll see it by every story all over. I'd recommend installing it, as you can just use prebuilt packages and don't have to install from source, which is possible but not trivial.

Relational Database (MySQL, PostgreSQL)

You will want to have your database set up before you begin installation. Fabric can install both PostgreSQL and MongoDB for you, but only on Ubuntu. Mac OS X users will want to have MySQL or PostgreSQL already installed. You can download MySQL or download PostgreSQL. Additionally, if running as a development machine on Mac OS X, I would recommend using MySQL with Sequel Pro as a GUI.

If you are installing MySQL, you will also need the MySQLDB python library:

sudo easy_install mysql-python

Fabric

Both Mac OS X and Linux require Fabric to be installed. Many common tasks, such as installing dependencies, deploying servers, migrations, and configurations are in fabfile.py.

sudo easy_install fabric

On recent installations of Mac OS X using XCode 4, you may run into issues around the ppc architecture. To fix this, simply run:

sudo ln -s /Developer/Platforms/iPhoneOS.platform/Developer/usr/libexec/gcc/darwin/ppc \
/Developer/usr/libexec/gcc/darwin
sudo ln -s /Developer/Platforms/iPhoneOS.platform/Developer/usr/libexec/gcc/darwin/ppc \
/usr/libexec/gcc/darwin

Sym-linking the ppc architecture comes from this StackOverflow answer on "assembler for architecture ppc not installed on Mac OS".

MongoDB

On top of MySQL/PostgreSQL, NewsBlur uses MongoDB to store non-relational data. You will want to download MongoDB. If you are on Ubuntu, the setup_mongo Fabric command will automatically do this for you, but Mac OS X needs to have it installed manually.

Numpy and Scipy

Not the easiest to get installed. If you are running Mac OS X, you have a few options:

Jammit

You must have Java 7 installed to run Jammit.

Other assorted packages

From inside the repository, run:

pip install -r requirements.txt

Configure paths

In fabfile.py there are two paths that need to be configured.

  • env.paths.NEWSBLUR is the relative path to the NewsBlur repository.
  • env.paths.VENDOR is the relative path to where all downloaded code should go.

In local_settings.py there are a few paths that need to be configured. Configure these after the installation below.

Installing on Mac OS X

  1. Using Mac OS X as a development environment, you can run all three servers (app, db, task) on the same system. You should have Fabric installed to run the fabfile.py. You should also have MySQL/PostgreSQL and MongoDB already installed.

    fab -R local setup_python
    fab -R local setup_mongoengine
    fab -R local setup_forked_mongoengine
    fab -R local setup_repo_local_settings
    fab -R local compress_assets
    

    If any of the packages fail to install (lxml, for instance), look through fabfile.py and check if there is a function that can be used to circumvent broken easy_install processes. For example, lxml may need libxslt and libxml2 to be installed. This is automated with the following Fabric command:

    fab -R local setup_libxml_code
    
  2. Configure MySQL/PostgreSQL by adding in a newsblur user and a newsblur database. Here's an example for MySQL:

    mysql_install_db --verbose --user=`whoami` --basedir="$(brew --prefix mysql)" --datadir=/path/to/var/mysql --tmpdir=/tmp
    mysql.server start
    mysql -u root
    > CREATE USER 'newsblur'@'localhost' IDENTIFIED BY '';
    > GRANT ALL PRIVILEGES ON *.* TO 'newsblur'@'localhost' WITH GRANT OPTION;
    > CREATE DATABASE newsblur;
    > exit
    

    Then load up the database with empty NewsBlur tables and bootstrap the database:

    ./manage.py syncdb --all --noinput
    ./manage.py migrate --fake
    ./manage.py migrate
    ./manage.py loaddata config/fixtures/bootstrap.json
    
  3. Start mongodb (if not already running):

    mongod run
    
  4. Run the development server. At this point, all dependencies should be installed and no additional configuration is needed. If you find that something is not working at this point, please email the resulting output to Samuel Clay at [email protected].

    ./manage.py runserver
    
  5. Navigate to:

     http://localhost:8000/ 
    

    Create an account. At the end of the account creation process, you will be redirected to https://localhost/profile/stripe_form. Hit the back button a few times, and you will be inside the app.

Installing on Linux / Ubuntu

If you are on Ubuntu, you can simply use Fabric to install NewsBlur and its many components. NewsBlur is designed to run on three separate servers: an app server, a db server, and assorted task servers. To install everything on a single machine, read through fabfile.py and setup all three servers (app, db, and task) without repeating the setup_common steps.

Finishing Installation

You must perform a few tasks to tie all of the various systems together.

  1. First, copy local_settings.py and fill in your OAuth keys, S3 keys, database names (if not newsblur), task server/broker address (RabbitMQ), and paths:

    cp local_settings.py.template local_settings.py
    

    Edit local_settings.py to change any keys that you have.

  2. Create the newsblur database in MySQL/PostgreSQL

    MySQL/PostgreSQL

    ./manage.py migrate
    

App server

fab -R local setup_app

Database server

fab -R local setup_db

Task server

fab -R local setup_task

Keeping NewsBlur Running

These commands keep NewsBlur fresh and updated. While on a development server, these commands do not need to be run more than once. However, you will probably want to run the refresh_feeds command regularly so you have new stories to test with and read.

Fetching feeds

If you just want to fetch feeds once, you can use the refresh_feeds management command:

./manage.py refresh_feeds

If you want to fetch feeds regardless of when they were last updated:

./manage.py refresh_feeds --force

You can also fetch the feeds for a specific user:

./manage.py refresh_feeds --user=newsblur

You'll want to put this refresh_feeds command on a timer to keep your feeds up to date.

Feedback

To populate the feedback table on the homepage, use the collect_feedback management command every few minutes:

./manage.py collect_feedback

Statistics

To populate the statistics graphs on the homepage, use the collect_stats management command every few minutes:

./manage.py collect_stats

Bootstrapping Search

Once you have an elasticsearch server running, you'll want to bootstrap it with feed and story indexes.

./manage.py index_feeds

Stories will be indexed automatically.

If you need to move search servers and want to just delete everything in the search database, you need to reset the MUserSearch table.

>>> from apps.search.models import MUserSearch
>>> MUserSearch.remove_all()

Running unit and integration tests

NewsBlur comes complete with a test suite that tests the functionality of the rss_feeds, reader, and feed importer. To run the test suite:

./manage.py test --settings=utils.test_settings

In Case of Downtime

You got the downtime message either through email or SMS. This is the order of operations for determining what's wrong.

0a. If downtime goes over 5 minutes, go to Twitter and say you're handling it. Be transparent about what it is, NewsBlur's followers are largely technical. Also the 502 page points users to Twitter for status updates.

0b. Ensure you have secrets-newsblur/configs/hosts installed in your /etc/hosts so server hostnames work.

  1. Check www.newsblur.com to confirm it's down.

    If you don't get a 502 page, then NewsBlur isn't even reachable and you just need to contact the hosting provider and yell at them.

  2. Check which servers can't be reached on HAProxy stats page. Basic auth can be found in secrets/configs/haproxy.conf. Search the secrets repo for "gimmiestats".

    Typically it'll be mongo, but any of the redis or postgres servers can be unreachable due to acts of god. Otherwise, a frequent cause is lack of disk space. There are monitors on every DB server watching for disk space, emailing me when they're running low, but it still happens.

  3. Check Sentry and see if the answer is at the top of the list.

    This will show if a database (redis, mongo, postgres) can't be found.

  4. Check the various databases:

    a. If Redis server (db_redis, db_redis_story, db_redis_pubsub) can't connect, redis is probably down.

    SSH into the offending server (or just check both the `db_redis` and `db_redis_story` servers) and
    check if `redis` is running. You can often `tail -f -n 100 /var/log/redis.log` to find out if
    background saving was being SIG(TERM|INT)'ed. When redis goes down, it's always because it's
    consuming too much memory. That shouldn't happen, so check the [munin
    graphs](http://db_redis/munin/).
    
    Boot it with `sudo /etc/init.d/redis start`.
    

    b. If mongo (db_mongo) can't connect, mongo is probably down.

    This is rare and usually signifies hardware failure. SSH into `db_mongo` and check logs with `tail
    -f -n 100 /var/log/mongodb/mongodb.log`. Start mongo with `sudo /etc/init.d/mongodb start` then
    promote the next largest mongodb server. You want to then promote one of the secondaries to
    primary, kill the offending primary machine, and rebuild it (preferably at a higher size). I
    recommend waiting a day to rebuild it so that you get a different machine. Don't forget to lodge a
    support ticket with the hosting provider so they know to check the machine.
    
    If it's the db_mongo_analytics machine, there is no backup nor secondaries of the data (because
    it's ephemeral and used for, you guessed it, analytics). You can easily provision a new mongodb
    server and point to that machine.
    
    If mongo is out of space, which happens, the servers need to be re-synced every 2-3 months to 
    compress the data bloat. Simply `rm -fr /var/lib/mongodb/*` and re-start Mongo. It will re-sync.
    
    If both secondaries are down, then the primary Mongo will go down. You'll need a secondary mongo
    in the sync state at the very least before the primary will accept reads. It shouldn't take long to
    get into that state, but you'll need a mongodb machine setup. You can immediately reuse the 
    non-working secondary if disk space is the only issue.
    

    c. If postgresql (db_pgsql) can't connect, postgres is probably down.

    This is the rarest of the rare and has in fact never happened. Machine failure. If you can salvage
    the db data, move it to another machine. Worst case you have nightly backups in S3. The fabfile.py
    has commands to assist in restoring from backup (the backup file just needs to be local).
    
  5. Point to a new/different machine

    a. Confirm the IP address of the new machine with fab list_do.

    b. Change secrets-newsbur/config/hosts to reflect the new machine.

    c. Copy the new hosts file to all machines with:

    fab all setup_hosts
    

    d. Changes should be instant, but you can also bounce every machine with:

    fab web deploy
    fab task celery
    

    e. Monitor utils/tlnb.py and utils/tlnbt.py for lots of reading and feed fetching.

  6. If feeds aren't fetching, check that the tasked_feeds queue is empty. You can drain it by running:

```
Feed.drain_task_feeds()
```

This happens when a deploy on the task servers hits faults and the task servers lose their 
connection without giving the tasked feeds back to the queue. Feeds that fall through this 
crack are automatically fixed after 24 hours, but if many feeds fall through due to a bad 
deploy or electrical failure, you'll want to accelerate that check by just draining the 
tasked feeds pool, adding those feeds back into the queue. This command is idempotent.

Author

License

NewsBlur is licensed under the MIT License. (See LICENSE)

Owner
Samuel Clay
Founder of NewsBlur and Turn Touch
Samuel Clay
Simple kivy project to help new kivy users build android apps with python.

Kivy Calculator A Simple Calculator made with kivy framework.Works on all platforms from Windows/linux to android. Description Simple kivy project to

Oussama Ben Sassi 6 Oct 06, 2022
Project Interface For nextcord-ext

Project Interface For nextcord-ext

nextcord-ext 1 Nov 13, 2021
Automated rop chain generation

This is the accompanying code to the blog post talking about automated rop chain generation. Build the test file with: make Install the dependencies:

Christopher Roberts 14 Nov 22, 2022
This package tries to emulate the behaviour of syntax proposed in PEP 671 via a decorator

Late-Bound Arguments This package tries to emulate the behaviour of syntax proposed in PEP 671 via a decorator. Usage Mention the names of the argumen

Shakya Majumdar 0 Feb 06, 2022
Python implementation of an automatic parallel parking system in a virtual environment, including path planning, path tracking, and parallel parking

Automatic Parallel Parking: Path Planning, Path Tracking & Control This repository contains a python implementation of an automatic parallel parking s

134 Jan 09, 2023
An After Effects render queue for ShotGrid Toolkit.

AEQueue An After Effects render queue for ShotGrid Toolkit. Features Render multiple comps to locations defined by templates in your Toolkit config. C

Brand New School 5 Nov 20, 2022
Машинное обучение на ФКН ВШЭ

Курс "Машинное обучение" на ФКН ВШЭ Конспекты лекций, материалы семинаров и домашние задания (теоретические, практические, соревнования) по курсу "Маш

Evgeny Sokolov 2.2k Jan 04, 2023
objectfactory is a python package to easily implement the factory design pattern for object creation, serialization, and polymorphism

py-object-factory objectfactory is a python package to easily implement the factory design pattern for object creation, serialization, and polymorphis

Devin A. Conley 6 Dec 14, 2022
Manually Install Python 2.7 pip without any problem !

Python2.7_install_pip Manually Install Python 2.7 pip without any problem ! Download installPip.py to your system and Run the code using this Command

Ali Jafari 1 Dec 09, 2021
⚙️ Compile, Read and update your .conf file in python

⚙️ Compile, Read and update your .conf file in python

Reece Harris 2 Aug 15, 2022
Scalene: a high-performance, high-precision CPU, GPU, and memory profiler for Python

Scalene: a high-performance CPU, GPU and memory profiler for Python by Emery Berger, Sam Stern, and Juan Altmayer Pizzorno. Scalene community Slack Ab

PLASMA @ UMass 7k Dec 30, 2022
Nook is a simple, concatenative programming language written in Python.

Nook Nook is a simple, concatenative programming language written in Python. Status Nook is currently WIP. It lacks a lot of basic feature, and will n

Wumi4 4 Jul 20, 2022
Create or join a private chatroom without any third-party middlemen in less than 30 seconds, available through an AES encrypted password protected link.

PY-CHAT Create or join a private chatroom without any third-party middlemen in less than 30 seconds, available through an AES encrypted password prote

1 Nov 24, 2021
A cheat sheet for streamlit

Streamlit Cheat Sheet App to summarise streamlit docs v1.0.0 There is also an accompanying png and pdf version https://github.com/daniellewisDL/stream

Daniel Lewis 221 Jan 04, 2023
The fundamentals of Python!

The fundamentals of Python Author: Mohamed NIANG, Staff ML Scientist Presentation This repository contains notebooks on the fundamentals of Python. Th

Mohamed NIANG 1 Mar 15, 2022
Ice Skating Simulator for Winter and Christmas [yay]

Ice Skating Simulator for Winter and Christmas [yay]

1 Aug 21, 2022
This is a backport of the BaseExceptionGroup and ExceptionGroup classes from Python 3.11.

This is a backport of the BaseExceptionGroup and ExceptionGroup classes from Python 3.11. It contains the following: The exceptiongroup.BaseExceptionG

Alex Grönholm 19 Dec 15, 2022
Checking-For-Fibonacci-Syquence-In-Python - Checking For Fibonacci Syquence In Python

Checking-For-Fibonacci-Syquence-In-Python The Fibonacci sequence is a set of num

John Michael Oliba 1 Feb 14, 2022
RELATE is an Environment for Learning And TEaching

RELATE Relate is an Environment for Learning And TEaching RELATE is a web-based courseware package. It is set apart by the following features: Focus o

Andreas Klöckner 311 Dec 25, 2022
Here You will Find CodeChef Challenge Solutions

Here You will Find CodeChef Challenge Solutions

kanishk kashyap 1 Sep 03, 2022