├── log └── .keep ├── storage └── .keep ├── tmp └── .keep ├── vendor └── .keep ├── lib ├── assets │ └── .keep └── tasks │ ├── .keep │ ├── elasticsearch.rake │ ├── jobs.rake │ ├── interim.rake │ ├── page_ids.rake │ ├── core_advertisers.rake │ ├── topics.rake │ ├── payers_and_advertisers.rake │ ├── swing_state_ads.rake │ └── ad_texts.rake ├── public ├── favicon.ico ├── apple-touch-icon.png ├── apple-touch-icon-precomposed.png ├── robots.txt ├── 500.html ├── 422.html └── 404.html ├── test ├── helpers │ └── .keep ├── mailers │ └── .keep ├── models │ ├── .keep │ ├── payer_test.rb │ ├── topic_test.rb │ ├── advertiser_test.rb │ ├── impression_test.rb │ ├── writable_ad_test.rb │ └── topic_writable_ad_test.rb ├── system │ └── .keep ├── controllers │ ├── .keep │ ├── ads_controller_test.rb │ ├── page_controller_test.rb │ ├── payers_controller_test.rb │ └── advertisers_controller_test.rb ├── fixtures │ ├── .keep │ ├── files │ │ └── .keep │ ├── topics.yml │ ├── payers.yml │ ├── topic_writable_ads.yml │ ├── writable_ads.yml │ ├── advertisers.yml │ └── impressions.yml ├── integration │ └── .keep ├── application_system_test_case.rb ├── channels │ └── application_cable │ │ └── connection_test.rb └── test_helper.rb ├── .ruby-version ├── app ├── assets │ ├── images │ │ └── .keep │ ├── config │ │ └── manifest.js │ └── stylesheets │ │ ├── ads.scss │ │ ├── page.scss │ │ ├── payers.scss │ │ ├── advertisers.scss │ │ ├── writable_pages.scss │ │ └── application.css ├── models │ ├── concerns │ │ └── .keep │ ├── job_run.rb │ ├── job.rb │ ├── application_record.rb │ ├── impressions_record.rb │ ├── ad_topic.rb │ ├── topic.rb │ ├── user.rb │ ├── writable_page.rb │ ├── big_spender.rb │ ├── ad_archive_report_page.rb │ ├── ad_archive_report.rb │ ├── writable_ad.rb │ ├── payer.rb │ ├── page.rb │ ├── ad_text.rb │ ├── ad.rb │ └── fbpac_ad.rb ├── controllers │ ├── concerns │ │ └── .keep │ ├── application_controller.rb │ ├── writable_pages_controller.rb │ ├── users │ │ ├── sessions_controller.rb │ │ ├── unlocks_controller.rb │ │ ├── confirmations_controller.rb │ │ ├── passwords_controller.rb │ │ ├── omniauth_callbacks_controller.rb │ │ └── registrations_controller.rb │ ├── payers_controller.rb │ ├── pages_controller.rb │ └── youtube_controller.rb ├── views │ ├── layouts │ │ ├── mailer.text.erb │ │ ├── mailer.html.erb │ │ └── application.html.erb │ ├── writable_pages │ │ └── update.html.erb │ ├── users │ │ ├── mailer │ │ │ ├── password_change.html.erb │ │ │ ├── confirmation_instructions.html.erb │ │ │ ├── unlock_instructions.html.erb │ │ │ ├── email_changed.html.erb │ │ │ └── reset_password_instructions.html.erb │ │ ├── registrations │ │ │ ├── index.html.erb │ │ │ ├── new.html.erb │ │ │ ├── new_other.html.erb │ │ │ └── edit.html.erb │ │ ├── shared │ │ │ ├── _error_messages.html.erb │ │ │ └── _links.html.erb │ │ ├── unlocks │ │ │ └── new.html.erb │ │ ├── passwords │ │ │ ├── new.html.erb │ │ │ └── edit.html.erb │ │ ├── confirmations │ │ │ └── new.html.erb │ │ └── sessions │ │ │ └── new.html.erb │ ├── fbpac_ads │ │ ├── suppress.html.erb │ │ └── pivot.html.erb │ ├── pages │ │ └── bigspenders.html.erb │ ├── ads │ │ ├── _ad.html.erb │ │ ├── swing_state_ads.html.erb │ │ └── overview.html.erb │ └── youtube │ │ ├── index.html.erb │ │ └── list.html.erb ├── helpers │ ├── ads_helper.rb │ ├── pages_helper.rb │ ├── payers_helper.rb │ ├── advertisers_helper.rb │ ├── application_helper.rb │ └── writable_pages_helper.rb ├── channels │ └── application_cable │ │ ├── channel.rb │ │ └── connection.rb ├── mailers │ └── application_mailer.rb ├── javascript │ ├── channels │ │ ├── index.js │ │ └── consumer.js │ └── packs │ │ └── application.js └── jobs │ └── application_job.rb ├── .browserslistrc ├── .python-version ├── config ├── webpack │ ├── environment.js │ ├── test.js │ ├── production.js │ └── development.js ├── spring.rb ├── environment.rb ├── initializers │ ├── mime_types.rb │ ├── filter_parameter_logging.rb │ ├── application_controller_renderer.rb │ ├── cookies_serializer.rb │ ├── backtrace_silencers.rb │ ├── wrap_parameters.rb │ ├── assets.rb │ ├── inflections.rb │ └── content_security_policy.rb ├── boot.rb ├── cable.yml ├── credentials.yml.enc ├── database.yml ├── locales │ ├── en.yml │ └── devise.en.yml ├── storage.yml ├── application.rb ├── puma.rb ├── environments │ ├── test.rb │ ├── development.rb │ └── production.rb ├── routes.rb └── webpacker.yml ├── config.ru ├── db ├── migrate │ ├── 20200421144120_add_page_id_to_fbpac_ads.rb │ ├── 20200108181722_add_ad_id_to_writable_ads.rb │ ├── 20200424154328_add_s3_url_to_writable_ads.rb │ ├── 20200421152421_add_page_id_to_writable_ads.rb │ ├── 20200421170805_add_page_id_index_to_writable_ads.rb │ ├── 20191203155349_add_disclaimer_to_writable_pages.rb │ ├── 20200423142347_add_core_to_writable_page.rb │ ├── 20200423144433_add_swing_state_ad_to_writable_ads.rb │ ├── 20200429161201_add_partisanship_to_writable_page.rb │ ├── 20200205183726_add_unaccent_to_postgres.rb │ ├── 20191004151713_create_topics.rb │ ├── 20200423194132_add_swing_states_to_writable_ads.rb │ ├── 20200108151338_add_text_hash_to_writable_ads.rb │ ├── 20200205171134_add_ad_text_id_to_ad_topics.rb │ ├── 20191004013753_create_payers.rb │ ├── 20191008172936_create_writable_pages.rb │ ├── 20200103144941_change_fbpac_ads_id_to_string.rb │ ├── 20200505211341_add_ad_texts_index_first_seen_desc_text_hash.rb │ ├── 20191216225447_add_amount_spent_since_start_date_to_ad_archive_report_page.rb │ ├── 20200108145543_create_ad_texts.rb │ ├── 20191004151746_create_ad_topics.rb │ ├── 20191020005347_ad_tranche_stuff_to_ad_archive_report_pages.rb │ ├── 20200508164400_create_jobs.rb │ ├── 20191023190103_add_index_to_ad_archive_report_pages.rb │ ├── 20200508164401_create_job_runs.rb │ ├── 20200508174349_add_aar_id_page_id_disclaimer_index_to_ad_archive_report_pages.rb │ ├── 20191017012957_create_ad_archive_reports.rb │ ├── 20191004145903_create_writable_ads.rb │ ├── 20200417210623_add_fields_to_ad_texts.rb │ ├── 20200420000000_add_indexes_to_ad_texts.rb │ ├── 20191017013045_create_ad_archive_report_pages.rb │ ├── 20191028150654_create_big_spenders.rb │ ├── 20200205171023_add_search_text_to_ad_texts.rb │ └── 20191106220234_devise_create_users.rb ├── seeds.rb └── schema.rb ├── Rakefile ├── bin ├── rake ├── rails ├── yarn ├── webpack ├── webpack-dev-server ├── spring ├── setup └── bundle ├── postcss.config.js ├── README.md ├── package.json ├── appspec.yml ├── LICENSE ├── .gitignore ├── babel.config.js ├── Gemfile ├── Gemfile.lock └── core_advertisers_20200422.csv /log/.keep: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /storage/.keep: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tmp/.keep: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /vendor/.keep: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /lib/assets/.keep: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /lib/tasks/.keep: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /public/favicon.ico: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /test/helpers/.keep: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /test/mailers/.keep: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /test/models/.keep: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /test/system/.keep: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /.ruby-version: -------------------------------------------------------------------------------- 1 | 2.6.5 2 | -------------------------------------------------------------------------------- /app/assets/images/.keep: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /test/controllers/.keep: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /test/fixtures/.keep: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /test/integration/.keep: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /.browserslistrc: -------------------------------------------------------------------------------- 1 | defaults 2 | -------------------------------------------------------------------------------- /.python-version: -------------------------------------------------------------------------------- 1 | 3.8.0 2 | -------------------------------------------------------------------------------- /app/models/concerns/.keep: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /public/apple-touch-icon.png: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /test/fixtures/files/.keep: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /app/controllers/concerns/.keep: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /public/apple-touch-icon-precomposed.png: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /app/views/layouts/mailer.text.erb: -------------------------------------------------------------------------------- 1 | <%= yield %> 2 | -------------------------------------------------------------------------------- /app/helpers/ads_helper.rb: -------------------------------------------------------------------------------- 1 | module AdsHelper 2 | end 3 | -------------------------------------------------------------------------------- /app/helpers/pages_helper.rb: -------------------------------------------------------------------------------- 1 | module PagesHelper 2 | end 3 | -------------------------------------------------------------------------------- /app/helpers/payers_helper.rb: -------------------------------------------------------------------------------- 1 | module PayersHelper 2 | end 3 | -------------------------------------------------------------------------------- /app/helpers/advertisers_helper.rb: -------------------------------------------------------------------------------- 1 | module AdvertisersHelper 2 | end 3 | -------------------------------------------------------------------------------- /app/helpers/application_helper.rb: -------------------------------------------------------------------------------- 1 | module ApplicationHelper 2 | end 3 | -------------------------------------------------------------------------------- /app/helpers/writable_pages_helper.rb: -------------------------------------------------------------------------------- 1 | module WritablePagesHelper 2 | end 3 | -------------------------------------------------------------------------------- /app/models/job_run.rb: -------------------------------------------------------------------------------- 1 | class JobRun < ApplicationRecord 2 | belongs_to :job 3 | end 4 | -------------------------------------------------------------------------------- /app/models/job.rb: -------------------------------------------------------------------------------- 1 | class Job < ApplicationRecord 2 | has_many :job_runs 3 | end 4 | 5 | -------------------------------------------------------------------------------- /app/assets/config/manifest.js: -------------------------------------------------------------------------------- 1 | //= link_tree ../images 2 | //= link_directory ../stylesheets .css 3 | -------------------------------------------------------------------------------- /app/models/application_record.rb: -------------------------------------------------------------------------------- 1 | class ApplicationRecord < ActiveRecord::Base 2 | self.abstract_class = true 3 | end 4 | -------------------------------------------------------------------------------- /config/webpack/environment.js: -------------------------------------------------------------------------------- 1 | const { environment } = require('@rails/webpacker') 2 | 3 | module.exports = environment 4 | -------------------------------------------------------------------------------- /public/robots.txt: -------------------------------------------------------------------------------- 1 | # See https://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file 2 | -------------------------------------------------------------------------------- /app/models/impressions_record.rb: -------------------------------------------------------------------------------- 1 | class ImpressionsRecord < ApplicationRecord 2 | self.table_name = "impressions" 3 | end 4 | -------------------------------------------------------------------------------- /app/channels/application_cable/channel.rb: -------------------------------------------------------------------------------- 1 | module ApplicationCable 2 | class Channel < ActionCable::Channel::Base 3 | end 4 | end 5 | -------------------------------------------------------------------------------- /app/views/writable_pages/update.html.erb: -------------------------------------------------------------------------------- 1 |

WritablePages#update

2 |

Find me in app/views/writable_pages/update.html.erb

3 | -------------------------------------------------------------------------------- /config/spring.rb: -------------------------------------------------------------------------------- 1 | Spring.watch( 2 | ".ruby-version", 3 | ".rbenv-vars", 4 | "tmp/restart.txt", 5 | "tmp/caching-dev.txt" 6 | ) 7 | -------------------------------------------------------------------------------- /app/channels/application_cable/connection.rb: -------------------------------------------------------------------------------- 1 | module ApplicationCable 2 | class Connection < ActionCable::Connection::Base 3 | end 4 | end 5 | -------------------------------------------------------------------------------- /app/mailers/application_mailer.rb: -------------------------------------------------------------------------------- 1 | class ApplicationMailer < ActionMailer::Base 2 | default from: 'from@example.com' 3 | layout 'mailer' 4 | end 5 | -------------------------------------------------------------------------------- /app/models/ad_topic.rb: -------------------------------------------------------------------------------- 1 | class AdTopic < ApplicationRecord 2 | belongs_to :topic 3 | belongs_to :ad_text 4 | 5 | def as_json 6 | topic 7 | end 8 | end 9 | -------------------------------------------------------------------------------- /config.ru: -------------------------------------------------------------------------------- 1 | # This file is used by Rack-based servers to start the application. 2 | 3 | require_relative 'config/environment' 4 | 5 | run Rails.application 6 | -------------------------------------------------------------------------------- /lib/tasks/elasticsearch.rake: -------------------------------------------------------------------------------- 1 | require 'elasticsearch/rails/tasks/import' 2 | 3 | 4 | # rake environment elasticsearch:import:model CLASS='FbpacAd' FORCE=y 5 | -------------------------------------------------------------------------------- /config/environment.rb: -------------------------------------------------------------------------------- 1 | # Load the Rails application. 2 | require_relative 'application' 3 | 4 | # Initialize the Rails application. 5 | Rails.application.initialize! 6 | -------------------------------------------------------------------------------- /test/models/payer_test.rb: -------------------------------------------------------------------------------- 1 | require 'test_helper' 2 | 3 | class PayerTest < ActiveSupport::TestCase 4 | # test "the truth" do 5 | # assert true 6 | # end 7 | end 8 | -------------------------------------------------------------------------------- /test/models/topic_test.rb: -------------------------------------------------------------------------------- 1 | require 'test_helper' 2 | 3 | class TopicTest < ActiveSupport::TestCase 4 | # test "the truth" do 5 | # assert true 6 | # end 7 | end 8 | -------------------------------------------------------------------------------- /app/views/users/mailer/password_change.html.erb: -------------------------------------------------------------------------------- 1 |

Hello <%= @resource.email %>!

2 | 3 |

We're contacting you to notify you that your password has been changed.

4 | -------------------------------------------------------------------------------- /test/models/advertiser_test.rb: -------------------------------------------------------------------------------- 1 | require 'test_helper' 2 | 3 | class AdvertiserTest < ActiveSupport::TestCase 4 | # test "the truth" do 5 | # assert true 6 | # end 7 | end 8 | -------------------------------------------------------------------------------- /test/models/impression_test.rb: -------------------------------------------------------------------------------- 1 | require 'test_helper' 2 | 3 | class ImpressionTest < ActiveSupport::TestCase 4 | # test "the truth" do 5 | # assert true 6 | # end 7 | end 8 | -------------------------------------------------------------------------------- /test/models/writable_ad_test.rb: -------------------------------------------------------------------------------- 1 | require 'test_helper' 2 | 3 | class WritableAdTest < ActiveSupport::TestCase 4 | # test "the truth" do 5 | # assert true 6 | # end 7 | end 8 | -------------------------------------------------------------------------------- /test/fixtures/topics.yml: -------------------------------------------------------------------------------- 1 | # Read about fixtures at https://api.rubyonrails.org/classes/ActiveRecord/FixtureSet.html 2 | 3 | one: 4 | topic: MyString 5 | 6 | two: 7 | topic: MyString 8 | -------------------------------------------------------------------------------- /config/webpack/test.js: -------------------------------------------------------------------------------- 1 | process.env.NODE_ENV = process.env.NODE_ENV || 'development' 2 | 3 | const environment = require('./environment') 4 | 5 | module.exports = environment.toWebpackConfig() 6 | -------------------------------------------------------------------------------- /db/migrate/20200421144120_add_page_id_to_fbpac_ads.rb: -------------------------------------------------------------------------------- 1 | class AddPageIdToFbpacAds < ActiveRecord::Migration[6.0] 2 | def change 3 | add_column :fbpac_ads, :page_id, :bigint 4 | end 5 | end 6 | -------------------------------------------------------------------------------- /test/models/topic_writable_ad_test.rb: -------------------------------------------------------------------------------- 1 | require 'test_helper' 2 | 3 | class TopicWritableAdTest < ActiveSupport::TestCase 4 | # test "the truth" do 5 | # assert true 6 | # end 7 | end 8 | -------------------------------------------------------------------------------- /config/webpack/production.js: -------------------------------------------------------------------------------- 1 | process.env.NODE_ENV = process.env.NODE_ENV || 'production' 2 | 3 | const environment = require('./environment') 4 | 5 | module.exports = environment.toWebpackConfig() 6 | -------------------------------------------------------------------------------- /db/migrate/20200108181722_add_ad_id_to_writable_ads.rb: -------------------------------------------------------------------------------- 1 | class AddAdIdToWritableAds < ActiveRecord::Migration[6.0] 2 | def change 3 | add_column :writable_ads, :ad_id, :text 4 | end 5 | end 6 | -------------------------------------------------------------------------------- /db/migrate/20200424154328_add_s3_url_to_writable_ads.rb: -------------------------------------------------------------------------------- 1 | class AddS3UrlToWritableAds < ActiveRecord::Migration[6.0] 2 | def change 3 | add_column :writable_ads, :s3_url, :text 4 | end 5 | end 6 | -------------------------------------------------------------------------------- /test/controllers/ads_controller_test.rb: -------------------------------------------------------------------------------- 1 | require 'test_helper' 2 | 3 | class AdsControllerTest < ActionDispatch::IntegrationTest 4 | # test "the truth" do 5 | # assert true 6 | # end 7 | end 8 | -------------------------------------------------------------------------------- /config/initializers/mime_types.rb: -------------------------------------------------------------------------------- 1 | # Be sure to restart your server when you modify this file. 2 | 3 | # Add new mime types for use in respond_to blocks: 4 | # Mime::Type.register "text/richtext", :rtf 5 | -------------------------------------------------------------------------------- /config/webpack/development.js: -------------------------------------------------------------------------------- 1 | process.env.NODE_ENV = process.env.NODE_ENV || 'development' 2 | 3 | const environment = require('./environment') 4 | 5 | module.exports = environment.toWebpackConfig() 6 | -------------------------------------------------------------------------------- /db/migrate/20200421152421_add_page_id_to_writable_ads.rb: -------------------------------------------------------------------------------- 1 | class AddPageIdToWritableAds < ActiveRecord::Migration[6.0] 2 | def change 3 | add_column :writable_ads, :page_id, :bigint 4 | end 5 | end 6 | -------------------------------------------------------------------------------- /db/migrate/20200421170805_add_page_id_index_to_writable_ads.rb: -------------------------------------------------------------------------------- 1 | class AddPageIdIndexToWritableAds < ActiveRecord::Migration[6.0] 2 | def change 3 | add_index :writable_ads, :page_id 4 | end 5 | end 6 | -------------------------------------------------------------------------------- /test/controllers/page_controller_test.rb: -------------------------------------------------------------------------------- 1 | require 'test_helper' 2 | 3 | class PageControllerTest < ActionDispatch::IntegrationTest 4 | # test "the truth" do 5 | # assert true 6 | # end 7 | end 8 | -------------------------------------------------------------------------------- /test/controllers/payers_controller_test.rb: -------------------------------------------------------------------------------- 1 | require 'test_helper' 2 | 3 | class PayersControllerTest < ActionDispatch::IntegrationTest 4 | # test "the truth" do 5 | # assert true 6 | # end 7 | end 8 | -------------------------------------------------------------------------------- /test/application_system_test_case.rb: -------------------------------------------------------------------------------- 1 | require "test_helper" 2 | 3 | class ApplicationSystemTestCase < ActionDispatch::SystemTestCase 4 | driven_by :selenium, using: :chrome, screen_size: [1400, 1400] 5 | end 6 | -------------------------------------------------------------------------------- /app/assets/stylesheets/ads.scss: -------------------------------------------------------------------------------- 1 | // Place all the styles related to the Ads controller here. 2 | // They will automatically be included in application.css. 3 | // You can use Sass (SCSS) here: http://sass-lang.com/ 4 | -------------------------------------------------------------------------------- /test/controllers/advertisers_controller_test.rb: -------------------------------------------------------------------------------- 1 | require 'test_helper' 2 | 3 | class AdvertisersControllerTest < ActionDispatch::IntegrationTest 4 | # test "the truth" do 5 | # assert true 6 | # end 7 | end 8 | -------------------------------------------------------------------------------- /app/assets/stylesheets/page.scss: -------------------------------------------------------------------------------- 1 | // Place all the styles related to the Page controller here. 2 | // They will automatically be included in application.css. 3 | // You can use Sass (SCSS) here: http://sass-lang.com/ 4 | -------------------------------------------------------------------------------- /app/assets/stylesheets/payers.scss: -------------------------------------------------------------------------------- 1 | // Place all the styles related to the Payers controller here. 2 | // They will automatically be included in application.css. 3 | // You can use Sass (SCSS) here: http://sass-lang.com/ 4 | -------------------------------------------------------------------------------- /db/migrate/20191203155349_add_disclaimer_to_writable_pages.rb: -------------------------------------------------------------------------------- 1 | class AddDisclaimerToWritablePages < ActiveRecord::Migration[6.0] 2 | def change 3 | add_column :writable_pages, :disclaimer, :string 4 | end 5 | end 6 | -------------------------------------------------------------------------------- /db/migrate/20200423142347_add_core_to_writable_page.rb: -------------------------------------------------------------------------------- 1 | class AddCoreToWritablePage < ActiveRecord::Migration[6.0] 2 | def change 3 | add_column :writable_pages, :core, :boolean, default: false 4 | end 5 | end 6 | -------------------------------------------------------------------------------- /db/migrate/20200423144433_add_swing_state_ad_to_writable_ads.rb: -------------------------------------------------------------------------------- 1 | class AddSwingStateAdToWritableAds < ActiveRecord::Migration[6.0] 2 | def change 3 | add_column :writable_ads, :swing_state_ad, :boolean 4 | end 5 | end 6 | -------------------------------------------------------------------------------- /db/migrate/20200429161201_add_partisanship_to_writable_page.rb: -------------------------------------------------------------------------------- 1 | class AddPartisanshipToWritablePage < ActiveRecord::Migration[6.0] 2 | def change 3 | add_column :writable_pages, :partisanship, :string 4 | end 5 | end 6 | -------------------------------------------------------------------------------- /app/assets/stylesheets/advertisers.scss: -------------------------------------------------------------------------------- 1 | // Place all the styles related to the Advertisers controller here. 2 | // They will automatically be included in application.css. 3 | // You can use Sass (SCSS) here: http://sass-lang.com/ 4 | -------------------------------------------------------------------------------- /config/boot.rb: -------------------------------------------------------------------------------- 1 | ENV['BUNDLE_GEMFILE'] ||= File.expand_path('../Gemfile', __dir__) 2 | 3 | require 'bundler/setup' # Set up gems listed in the Gemfile. 4 | require 'bootsnap/setup' # Speed up boot time by caching expensive operations. 5 | -------------------------------------------------------------------------------- /db/migrate/20200205183726_add_unaccent_to_postgres.rb: -------------------------------------------------------------------------------- 1 | class AddUnaccentToPostgres < ActiveRecord::Migration[6.0] 2 | def up 3 | execute "create extension if not exists unaccent;" 4 | end 5 | def down 6 | 7 | end 8 | end 9 | -------------------------------------------------------------------------------- /test/fixtures/payers.yml: -------------------------------------------------------------------------------- 1 | # Read about fixtures at https://api.rubyonrails.org/classes/ActiveRecord/FixtureSet.html 2 | 3 | one: 4 | name: MyString 5 | notes: MyText 6 | 7 | two: 8 | name: MyString 9 | notes: MyText 10 | -------------------------------------------------------------------------------- /app/assets/stylesheets/writable_pages.scss: -------------------------------------------------------------------------------- 1 | // Place all the styles related to the WritablePages controller here. 2 | // They will automatically be included in application.css. 3 | // You can use Sass (SCSS) here: https://sass-lang.com/ 4 | -------------------------------------------------------------------------------- /db/migrate/20191004151713_create_topics.rb: -------------------------------------------------------------------------------- 1 | class CreateTopics < ActiveRecord::Migration[6.0] 2 | def change 3 | create_table :topics do |t| 4 | t.string :topic 5 | 6 | t.timestamps 7 | end 8 | end 9 | end 10 | -------------------------------------------------------------------------------- /db/migrate/20200423194132_add_swing_states_to_writable_ads.rb: -------------------------------------------------------------------------------- 1 | class AddSwingStatesToWritableAds < ActiveRecord::Migration[6.0] 2 | def change 3 | add_column :writable_ads, :states, :text, array: true, default: [] 4 | end 5 | end 6 | -------------------------------------------------------------------------------- /app/models/topic.rb: -------------------------------------------------------------------------------- 1 | class Topic < ApplicationRecord 2 | has_many :ads, through: :ad_topics 3 | has_many :ad_topics 4 | 5 | def as_json(options) 6 | super(options).without("created_at", "updated_at") 7 | end 8 | 9 | 10 | end 11 | -------------------------------------------------------------------------------- /config/initializers/filter_parameter_logging.rb: -------------------------------------------------------------------------------- 1 | # Be sure to restart your server when you modify this file. 2 | 3 | # Configure sensitive parameters which will be filtered from the log file. 4 | Rails.application.config.filter_parameters += [:password] 5 | -------------------------------------------------------------------------------- /db/migrate/20200108151338_add_text_hash_to_writable_ads.rb: -------------------------------------------------------------------------------- 1 | class AddTextHashToWritableAds < ActiveRecord::Migration[6.0] 2 | def change 3 | add_column :writable_ads, :text_hash, :string 4 | add_index :writable_ads, :text_hash 5 | end 6 | end 7 | -------------------------------------------------------------------------------- /Rakefile: -------------------------------------------------------------------------------- 1 | # Add your own tasks in files placed in lib/tasks ending in .rake, 2 | # for example lib/tasks/capistrano.rake, and they will automatically be available to Rake. 3 | 4 | require_relative 'config/application' 5 | 6 | Rails.application.load_tasks 7 | -------------------------------------------------------------------------------- /config/cable.yml: -------------------------------------------------------------------------------- 1 | development: 2 | adapter: async 3 | 4 | test: 5 | adapter: test 6 | 7 | production: 8 | adapter: redis 9 | url: <%= ENV.fetch("REDIS_URL") { "redis://localhost:6379/1" } %> 10 | channel_prefix: AtiDashboard_production 11 | -------------------------------------------------------------------------------- /db/migrate/20200205171134_add_ad_text_id_to_ad_topics.rb: -------------------------------------------------------------------------------- 1 | class AddAdTextIdToAdTopics < ActiveRecord::Migration[6.0] 2 | def change 3 | add_column :ad_topics, :ad_text_id, :integer 4 | remove_column :ad_topics, :archive_id, :bigint 5 | end 6 | end 7 | -------------------------------------------------------------------------------- /app/javascript/channels/index.js: -------------------------------------------------------------------------------- 1 | // Load all the channels within this directory and all subdirectories. 2 | // Channel files must be named *_channel.js. 3 | 4 | const channels = require.context('.', true, /_channel\.js$/) 5 | channels.keys().forEach(channels) 6 | -------------------------------------------------------------------------------- /bin/rake: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env ruby 2 | begin 3 | load File.expand_path('../spring', __FILE__) 4 | rescue LoadError => e 5 | raise unless e.message.include?('spring') 6 | end 7 | require_relative '../config/boot' 8 | require 'rake' 9 | Rake.application.run 10 | -------------------------------------------------------------------------------- /db/migrate/20191004013753_create_payers.rb: -------------------------------------------------------------------------------- 1 | class CreatePayers < ActiveRecord::Migration[6.0] 2 | def change 3 | create_table :payers do |t| 4 | t.string :name 5 | t.text :notes 6 | 7 | t.timestamps 8 | end 9 | end 10 | end 11 | -------------------------------------------------------------------------------- /app/views/users/mailer/confirmation_instructions.html.erb: -------------------------------------------------------------------------------- 1 |

Welcome <%= @email %>!

2 | 3 |

You can confirm your account email through the link below:

4 | 5 |

<%= link_to 'Confirm my account', confirmation_url(@resource, confirmation_token: @token) %>

6 | -------------------------------------------------------------------------------- /db/migrate/20191008172936_create_writable_pages.rb: -------------------------------------------------------------------------------- 1 | class CreateWritablePages < ActiveRecord::Migration[6.0] 2 | def change 3 | create_table :writable_pages do |t| 4 | t.bigint :page_id 5 | t.text :notes 6 | t.timestamps 7 | end 8 | end 9 | end 10 | -------------------------------------------------------------------------------- /db/migrate/20200103144941_change_fbpac_ads_id_to_string.rb: -------------------------------------------------------------------------------- 1 | class ChangeFbpacAdsIdToString < ActiveRecord::Migration[6.0] 2 | def up 3 | change_column :fbpac_ads, :id, :text 4 | end 5 | 6 | def down 7 | change_column :fbpac_ads, :id, :bigint 8 | end 9 | end 10 | -------------------------------------------------------------------------------- /db/migrate/20200505211341_add_ad_texts_index_first_seen_desc_text_hash.rb: -------------------------------------------------------------------------------- 1 | class AddAdTextsIndexFirstSeenDescTextHash < ActiveRecord::Migration[6.0] 2 | def change 3 | add_index :ad_texts, [:first_seen, :text_hash], order: {first_seen: :desc, text_hash: :asc} 4 | end 5 | end 6 | -------------------------------------------------------------------------------- /test/fixtures/topic_writable_ads.yml: -------------------------------------------------------------------------------- 1 | # Read about fixtures at https://api.rubyonrails.org/classes/ActiveRecord/FixtureSet.html 2 | 3 | one: 4 | writable_ad_id: 5 | topic_id: 6 | proportion: 1.5 7 | 8 | two: 9 | writable_ad_id: 10 | topic_id: 11 | proportion: 1.5 12 | -------------------------------------------------------------------------------- /config/initializers/application_controller_renderer.rb: -------------------------------------------------------------------------------- 1 | # Be sure to restart your server when you modify this file. 2 | 3 | # ActiveSupport::Reloader.to_prepare do 4 | # ApplicationController.renderer.defaults.merge!( 5 | # http_host: 'example.org', 6 | # https: false 7 | # ) 8 | # end 9 | -------------------------------------------------------------------------------- /db/migrate/20191216225447_add_amount_spent_since_start_date_to_ad_archive_report_page.rb: -------------------------------------------------------------------------------- 1 | class AddAmountSpentSinceStartDateToAdArchiveReportPage < ActiveRecord::Migration[6.0] 2 | def change 3 | add_column :ad_archive_report_pages, :amount_spent_since_start_date, :integer 4 | end 5 | end 6 | -------------------------------------------------------------------------------- /db/migrate/20200108145543_create_ad_texts.rb: -------------------------------------------------------------------------------- 1 | class CreateAdTexts < ActiveRecord::Migration[6.0] 2 | def change 3 | create_table :ad_texts do |t| 4 | t.text :text 5 | t.string :text_hash 6 | t.text :vec 7 | 8 | t.timestamps 9 | end 10 | end 11 | end 12 | -------------------------------------------------------------------------------- /postcss.config.js: -------------------------------------------------------------------------------- 1 | module.exports = { 2 | plugins: [ 3 | require('postcss-import'), 4 | require('postcss-flexbugs-fixes'), 5 | require('postcss-preset-env')({ 6 | autoprefixer: { 7 | flexbox: 'no-2009' 8 | }, 9 | stage: 3 10 | }) 11 | ] 12 | } 13 | -------------------------------------------------------------------------------- /config/initializers/cookies_serializer.rb: -------------------------------------------------------------------------------- 1 | # Be sure to restart your server when you modify this file. 2 | 3 | # Specify a serializer for the signed and encrypted cookie jars. 4 | # Valid options are :json, :marshal, and :hybrid. 5 | Rails.application.config.action_dispatch.cookies_serializer = :json 6 | -------------------------------------------------------------------------------- /app/models/user.rb: -------------------------------------------------------------------------------- 1 | class User < ApplicationRecord 2 | # Include default devise modules. Others available are: 3 | # :confirmable, :lockable, :timeoutable, :trackable and :omniauthable 4 | devise :database_authenticatable, 5 | :recoverable, :rememberable, :validatable, :registerable 6 | end 7 | -------------------------------------------------------------------------------- /app/models/writable_page.rb: -------------------------------------------------------------------------------- 1 | class WritablePage < ApplicationRecord 2 | belongs_to :page, primary_key: :page_id, foreign_key: :page_id, optional: true 3 | 4 | has_many :ad_texts, primary_key: :page_id, foreign_key: :page_id 5 | has_many :ads, primary_key: :page_id, foreign_key: :page_id 6 | 7 | end 8 | -------------------------------------------------------------------------------- /db/migrate/20191004151746_create_ad_topics.rb: -------------------------------------------------------------------------------- 1 | class CreateAdTopics < ActiveRecord::Migration[6.0] 2 | def change 3 | create_table :ad_topics do |t| 4 | t.bigint :archive_id 5 | t.integer :topic_id 6 | t.float :proportion 7 | 8 | t.timestamps 9 | end 10 | end 11 | end 12 | -------------------------------------------------------------------------------- /db/migrate/20191020005347_ad_tranche_stuff_to_ad_archive_report_pages.rb: -------------------------------------------------------------------------------- 1 | class AdTrancheStuffToAdArchiveReportPages < ActiveRecord::Migration[6.0] 2 | def change 3 | change_table :ad_archive_report_pages do |t| 4 | t.integer :ads_this_tranche 5 | t.integer :spend_this_tranche 6 | end 7 | end 8 | end 9 | -------------------------------------------------------------------------------- /db/migrate/20200508164400_create_jobs.rb: -------------------------------------------------------------------------------- 1 | class CreateJobs < ActiveRecord::Migration[6.0] 2 | def change 3 | create_table :jobs do |t| 4 | t.string :name 5 | t.integer :expected_run_rate 6 | t.numeric :estimated_duration 7 | 8 | t.timestamps 9 | end 10 | end 11 | end 12 | -------------------------------------------------------------------------------- /bin/rails: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env ruby 2 | begin 3 | load File.expand_path('../spring', __FILE__) 4 | rescue LoadError => e 5 | raise unless e.message.include?('spring') 6 | end 7 | APP_PATH = File.expand_path('../config/application', __dir__) 8 | require_relative '../config/boot' 9 | require 'rails/commands' 10 | -------------------------------------------------------------------------------- /app/javascript/channels/consumer.js: -------------------------------------------------------------------------------- 1 | // Action Cable provides the framework to deal with WebSockets in Rails. 2 | // You can generate new channels where WebSocket features live using the `rails generate channel` command. 3 | 4 | import { createConsumer } from "@rails/actioncable" 5 | 6 | export default createConsumer() 7 | -------------------------------------------------------------------------------- /app/jobs/application_job.rb: -------------------------------------------------------------------------------- 1 | class ApplicationJob < ActiveJob::Base 2 | # Automatically retry jobs that encountered a deadlock 3 | # retry_on ActiveRecord::Deadlocked 4 | 5 | # Most jobs are safe to ignore if the underlying records are no longer available 6 | # discard_on ActiveJob::DeserializationError 7 | end 8 | -------------------------------------------------------------------------------- /app/views/layouts/mailer.html.erb: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 8 | 9 | 10 | 11 | <%= yield %> 12 | 13 | 14 | -------------------------------------------------------------------------------- /db/migrate/20191023190103_add_index_to_ad_archive_report_pages.rb: -------------------------------------------------------------------------------- 1 | class AddIndexToAdArchiveReportPages < ActiveRecord::Migration[6.0] 2 | def change 3 | add_index :ad_archive_report_pages, [:ad_archive_report_id, :page_id], name: "index_ad_archive_report_pages_on_ad_archive_report_id_page_id" 4 | end 5 | 6 | end 7 | -------------------------------------------------------------------------------- /db/migrate/20200508164401_create_job_runs.rb: -------------------------------------------------------------------------------- 1 | class CreateJobRuns < ActiveRecord::Migration[6.0] 2 | def change 3 | create_table :job_runs do |t| 4 | t.integer :job_id 5 | t.datetime :start_time 6 | t.datetime :end_time 7 | t.boolean :success 8 | 9 | t.timestamps 10 | end 11 | end 12 | end 13 | -------------------------------------------------------------------------------- /db/migrate/20200508174349_add_aar_id_page_id_disclaimer_index_to_ad_archive_report_pages.rb: -------------------------------------------------------------------------------- 1 | class AddAarIdPageIdDisclaimerIndexToAdArchiveReportPages < ActiveRecord::Migration[6.0] 2 | def change 3 | add_index :ad_archive_report_pages, [:ad_archive_report_id, :page_id, :disclaimer], name: "index_aarps_aar_id_page_id_discl" 4 | end 5 | end 6 | -------------------------------------------------------------------------------- /app/controllers/application_controller.rb: -------------------------------------------------------------------------------- 1 | class ApplicationController < ActionController::Base 2 | before_action :authenticate_user! 3 | protect_from_forgery with: :exception 4 | 5 | 6 | protected 7 | 8 | def force_trailing_slash 9 | redirect_to request.original_url + '/' unless request.original_url.match(/\/$/) 10 | end 11 | end -------------------------------------------------------------------------------- /app/views/users/mailer/unlock_instructions.html.erb: -------------------------------------------------------------------------------- 1 |

Hello <%= @resource.email %>!

2 | 3 |

Your account has been locked due to an excessive number of unsuccessful sign in attempts.

4 | 5 |

Click the link below to unlock your account:

6 | 7 |

<%= link_to 'Unlock my account', unlock_url(@resource, unlock_token: @token) %>

8 | -------------------------------------------------------------------------------- /test/channels/application_cable/connection_test.rb: -------------------------------------------------------------------------------- 1 | require "test_helper" 2 | 3 | class ApplicationCable::ConnectionTest < ActionCable::Connection::TestCase 4 | # test "connects with cookies" do 5 | # cookies.signed[:user_id] = 42 6 | # 7 | # connect 8 | # 9 | # assert_equal connection.user_id, "42" 10 | # end 11 | end 12 | -------------------------------------------------------------------------------- /bin/yarn: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env ruby 2 | APP_ROOT = File.expand_path('..', __dir__) 3 | Dir.chdir(APP_ROOT) do 4 | begin 5 | exec "yarnpkg", *ARGV 6 | rescue Errno::ENOENT 7 | $stderr.puts "Yarn executable was not detected in the system." 8 | $stderr.puts "Download Yarn at https://yarnpkg.com/en/docs/install" 9 | exit 1 10 | end 11 | end 12 | -------------------------------------------------------------------------------- /db/migrate/20191017012957_create_ad_archive_reports.rb: -------------------------------------------------------------------------------- 1 | class CreateAdArchiveReports < ActiveRecord::Migration[6.0] 2 | def change 3 | create_table :ad_archive_reports do |t| 4 | t.datetime :scrape_date 5 | t.text :s3_url 6 | t.text :kind 7 | t.boolean :loaded, default: false 8 | 9 | t.timestamps 10 | end 11 | end 12 | end 13 | -------------------------------------------------------------------------------- /app/views/fbpac_ads/suppress.html.erb: -------------------------------------------------------------------------------- 1 |

Quartz Political Facebook Ads Dashboard

2 | 3 |

Suppress a false-positive actually-not-political ad

4 | 5 | <%= form_with(url: "/fbpac_ads/suppress", method: "post") do %> 6 | <%= label_tag(:ad_id, "Ad ID:") %> 7 | <%= text_field_tag(:ad_id) %> 8 | <%= submit_tag("Suppress") %> 9 | <% end %> 10 | -------------------------------------------------------------------------------- /app/views/users/mailer/email_changed.html.erb: -------------------------------------------------------------------------------- 1 |

Hello <%= @email %>!

2 | 3 | <% if @resource.try(:unconfirmed_email?) %> 4 |

We're contacting you to notify you that your email is being changed to <%= @resource.unconfirmed_email %>.

5 | <% else %> 6 |

We're contacting you to notify you that your email has been changed to <%= @resource.email %>.

7 | <% end %> 8 | -------------------------------------------------------------------------------- /db/migrate/20191004145903_create_writable_ads.rb: -------------------------------------------------------------------------------- 1 | class CreateWritableAds < ActiveRecord::Migration[6.0] 2 | def change 3 | create_table :writable_ads do |t| 4 | t.string :partisanship 5 | t.string :purpose 6 | t.string :optimism 7 | t.string :attack 8 | t.bigint :archive_id 9 | 10 | t.timestamps 11 | end 12 | end 13 | end 14 | -------------------------------------------------------------------------------- /db/migrate/20200417210623_add_fields_to_ad_texts.rb: -------------------------------------------------------------------------------- 1 | class AddFieldsToAdTexts < ActiveRecord::Migration[6.0] 2 | def change 3 | add_column :ad_texts, :page_id, :bigint 4 | add_column :ad_texts, :advertiser, :text 5 | add_column :ad_texts, :paid_for_by, :text 6 | add_column :ad_texts, :first_seen, :datetime 7 | add_column :ad_texts, :last_seen, :datetime 8 | end 9 | end 10 | -------------------------------------------------------------------------------- /test/fixtures/writable_ads.yml: -------------------------------------------------------------------------------- 1 | # Read about fixtures at https://api.rubyonrails.org/classes/ActiveRecord/FixtureSet.html 2 | 3 | one: 4 | partisanship: MyString 5 | purpose: MyString 6 | optimism: MyString 7 | attack: MyString 8 | ad_archive_id: 9 | 10 | two: 11 | partisanship: MyString 12 | purpose: MyString 13 | optimism: MyString 14 | attack: MyString 15 | ad_archive_id: 16 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Database for Facebook ads 2 | 3 | This is a repo for a dashboard for Facebook ads (and perhaps other online political content) built for reporters and researchers, by Quartz. 4 | 5 | If you're a reporter or researcher, email Jeremy Merrill at Quartz to get access. (Email is his first name at qz.com). 6 | 7 | --- 8 | ## frontend 9 | 10 | is here: https://github.com/Quartz/pol-ad-dashboard 11 | -------------------------------------------------------------------------------- /package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "AtiDashboard", 3 | "private": true, 4 | "dependencies": { 5 | "@rails/actioncable": "^6.0.2", 6 | "@rails/activestorage": "^6.0.2", 7 | "@rails/ujs": "^6.0.2", 8 | "@rails/webpacker": "^4.2.2", 9 | "turbolinks": "^5.2.0", 10 | "yarn": "^1.22.4" 11 | }, 12 | "version": "0.1.0", 13 | "devDependencies": { 14 | "webpack-dev-server": "^3.10.3" 15 | } 16 | } 17 | -------------------------------------------------------------------------------- /test/fixtures/advertisers.yml: -------------------------------------------------------------------------------- 1 | # Read about fixtures at https://api.rubyonrails.org/classes/ActiveRecord/FixtureSet.html 2 | 3 | # This model initially had no columns defined. If you add columns to the 4 | # model remove the '{}' from the fixture names and add the columns immediately 5 | # below each fixture, per the syntax in the comments below 6 | # 7 | one: {} 8 | # column: value 9 | # 10 | two: {} 11 | # column: value 12 | -------------------------------------------------------------------------------- /test/fixtures/impressions.yml: -------------------------------------------------------------------------------- 1 | # Read about fixtures at https://api.rubyonrails.org/classes/ActiveRecord/FixtureSet.html 2 | 3 | # This model initially had no columns defined. If you add columns to the 4 | # model remove the '{}' from the fixture names and add the columns immediately 5 | # below each fixture, per the syntax in the comments below 6 | # 7 | one: {} 8 | # column: value 9 | # 10 | two: {} 11 | # column: value 12 | -------------------------------------------------------------------------------- /db/migrate/20200420000000_add_indexes_to_ad_texts.rb: -------------------------------------------------------------------------------- 1 | class AddIndexesToAdTexts < ActiveRecord::Migration[6.0] 2 | def change 3 | add_index :ad_texts, :page_id 4 | add_index :ad_texts, :advertiser 5 | add_index :ad_texts, :paid_for_by 6 | add_index :ad_texts, :first_seen 7 | add_index :ad_texts, :last_seen 8 | add_index :writable_ads, :archive_id 9 | add_index :writable_ads, :ad_id 10 | end 11 | end 12 | -------------------------------------------------------------------------------- /app/views/users/registrations/index.html.erb: -------------------------------------------------------------------------------- 1 |

All users

2 | 3 | 4 | <% @users.each do |user| %> 5 | 6 | 7 | 8 | 9 | <% end %> 10 |
<%= user.email %><%= button_to "delete", delete_other_user_path(user.id), method: :delete, data: { confirm: 'Are you sure?' } %>
11 | 12 | create new user 13 | <%= render "users/shared/links" %> 14 | -------------------------------------------------------------------------------- /db/migrate/20191017013045_create_ad_archive_report_pages.rb: -------------------------------------------------------------------------------- 1 | class CreateAdArchiveReportPages < ActiveRecord::Migration[6.0] 2 | def change 3 | create_table :ad_archive_report_pages do |t| 4 | t.bigint :page_id 5 | t.string :page_name 6 | t.string :disclaimer 7 | t.integer :amount_spent 8 | t.integer :ads_count 9 | t.integer :ad_archive_report_id 10 | 11 | t.timestamps 12 | end 13 | end 14 | end 15 | -------------------------------------------------------------------------------- /test/test_helper.rb: -------------------------------------------------------------------------------- 1 | ENV['RAILS_ENV'] ||= 'test' 2 | require_relative '../config/environment' 3 | require 'rails/test_help' 4 | 5 | class ActiveSupport::TestCase 6 | # Run tests in parallel with specified workers 7 | parallelize(workers: :number_of_processors) 8 | 9 | # Setup all fixtures in test/fixtures/*.yml for all tests in alphabetical order. 10 | fixtures :all 11 | 12 | # Add more helper methods to be used by all tests here... 13 | end 14 | -------------------------------------------------------------------------------- /config/initializers/backtrace_silencers.rb: -------------------------------------------------------------------------------- 1 | # Be sure to restart your server when you modify this file. 2 | 3 | # You can add backtrace silencers for libraries that you're using but don't wish to see in your backtraces. 4 | # Rails.backtrace_cleaner.add_silencer { |line| line =~ /my_noisy_library/ } 5 | 6 | # You can also remove all the silencers if you're trying to debug a problem that might stem from framework code. 7 | # Rails.backtrace_cleaner.remove_silencers! 8 | -------------------------------------------------------------------------------- /app/controllers/writable_pages_controller.rb: -------------------------------------------------------------------------------- 1 | class WritablePagesController < ApplicationController 2 | def update 3 | @writable_page = WritablePage.find_or_initialize_by(page_id: params[:page_id], disclaimer: params[:disclaimer]) 4 | @writable_page.notes = params[:notes] 5 | @writable_page.save 6 | 7 | respond_to do |format| 8 | format.json { render json: { 9 | ok: "ok" 10 | } 11 | } 12 | end 13 | end 14 | end 15 | -------------------------------------------------------------------------------- /app/views/users/mailer/reset_password_instructions.html.erb: -------------------------------------------------------------------------------- 1 |

Hello <%= @resource.email %>!

2 | 3 |

Someone has requested a link to change your password. You can do this through the link below.

4 | 5 |

<%= link_to 'Change my password', edit_password_url(@resource, reset_password_token: @token) %>

6 | 7 |

If you didn't request this, please ignore this email.

8 |

Your password won't change until you access the link above and create a new one.

9 | -------------------------------------------------------------------------------- /config/credentials.yml.enc: -------------------------------------------------------------------------------- 1 | W19061mRJLxyZD1MyupVvkW+HoNVNYO6COYuMWwtFI2ZS3kJ4wg6KAsd9/cknstSL4SNIgkFMl6Btk/iaDJfIwAVvUAZi54gXV2EK1brFFjqNC9AFeBYNKk0DCWv0spm3FPR1kuh/vAydHyPgRfDvycSy7sCb3Ewng4IO72z38ZOd3wABQ/04x56y7VXh/i9H0ng2MP13kD2da57xJ7uBMG9z/99QBaLMeclYeh013bchIB88XBSwgkFK2UeoxfXug7kSW+hxebl4PA3hNy/n5fZRVB5gznERVz59xKmC2JIzk5F10IsuRwXp31f70covMNw9mfoIaODeLrPKkeVzgzZ5naanuK9N+V+jedOrRkW+f0aLs6gIISH0zCPxen3d/EcRnx9vb1o5bxf2xFojl7Ln6ZL8Yvj3Mej--x8PwAy1ean2RisDp--1X+9/yig6c2I5zlJwSc2mg== -------------------------------------------------------------------------------- /db/migrate/20191028150654_create_big_spenders.rb: -------------------------------------------------------------------------------- 1 | class CreateBigSpenders < ActiveRecord::Migration[6.0] 2 | def change 3 | create_table :big_spenders do |t| 4 | t.integer :ad_archive_report_id 5 | t.integer :previous_ad_archive_report_id 6 | t.integer :ad_archive_report_page_id 7 | t.bigint :page_id 8 | t.integer :spend_amount 9 | t.integer :duration_days 10 | t.boolean :is_new 11 | 12 | t.timestamps 13 | end 14 | end 15 | end 16 | -------------------------------------------------------------------------------- /app/views/users/shared/_error_messages.html.erb: -------------------------------------------------------------------------------- 1 | <% if resource.errors.any? %> 2 |
3 |

4 | <%= I18n.t("errors.messages.not_saved", 5 | count: resource.errors.count, 6 | resource: resource.class.model_name.human.downcase) 7 | %> 8 |

9 | 14 |
15 | <% end %> 16 | -------------------------------------------------------------------------------- /bin/webpack: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env ruby 2 | 3 | ENV["RAILS_ENV"] ||= ENV["RACK_ENV"] || "development" 4 | ENV["NODE_ENV"] ||= "development" 5 | 6 | require "pathname" 7 | ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../Gemfile", 8 | Pathname.new(__FILE__).realpath) 9 | 10 | require "rubygems" 11 | require "bundler/setup" 12 | 13 | require "webpacker" 14 | require "webpacker/webpack_runner" 15 | 16 | APP_ROOT = File.expand_path("..", __dir__) 17 | Dir.chdir(APP_ROOT) do 18 | Webpacker::WebpackRunner.run(ARGV) 19 | end 20 | -------------------------------------------------------------------------------- /bin/webpack-dev-server: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env ruby 2 | 3 | ENV["RAILS_ENV"] ||= ENV["RACK_ENV"] || "development" 4 | ENV["NODE_ENV"] ||= "development" 5 | 6 | require "pathname" 7 | ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../Gemfile", 8 | Pathname.new(__FILE__).realpath) 9 | 10 | require "rubygems" 11 | require "bundler/setup" 12 | 13 | require "webpacker" 14 | require "webpacker/dev_server_runner" 15 | 16 | APP_ROOT = File.expand_path("..", __dir__) 17 | Dir.chdir(APP_ROOT) do 18 | Webpacker::DevServerRunner.run(ARGV) 19 | end 20 | -------------------------------------------------------------------------------- /app/models/big_spender.rb: -------------------------------------------------------------------------------- 1 | class BigSpender < ApplicationRecord 2 | belongs_to :ad_archive_report 3 | belongs_to :page, optional: true 4 | belongs_to :ad_archive_report_page 5 | 6 | belongs_to :writable_page, optional: true, primary_key: :page_id, foreign_key: :page_id 7 | 8 | def to_s 9 | aarp = self.ad_archive_report_page 10 | "#{self.is_new? ? "NEW ADVERTISER" : ""} #{aarp.page_name} (#{aarp.disclaimer}) spent #{self.spend_amount.to_s.reverse.scan(/\d{3}|.+/).join(",").reverse} in the last #{self.duration_days} days".strip 11 | end 12 | 13 | end 14 | -------------------------------------------------------------------------------- /app/views/users/unlocks/new.html.erb: -------------------------------------------------------------------------------- 1 |

Resend unlock instructions

2 | 3 | <%= form_for(resource, as: resource_name, url: unlock_path(resource_name), html: { method: :post }) do |f| %> 4 | <%= render "devise/shared/error_messages", resource: resource %> 5 | 6 |
7 | <%= f.label :email %>
8 | <%= f.email_field :email, autofocus: true, autocomplete: "email" %> 9 |
10 | 11 |
12 | <%= f.submit "Resend unlock instructions" %> 13 |
14 | <% end %> 15 | 16 | <%= render "users/shared/links" %> 17 | -------------------------------------------------------------------------------- /config/initializers/wrap_parameters.rb: -------------------------------------------------------------------------------- 1 | # Be sure to restart your server when you modify this file. 2 | 3 | # This file contains settings for ActionController::ParamsWrapper which 4 | # is enabled by default. 5 | 6 | # Enable parameter wrapping for JSON. You can disable this by setting :format to an empty array. 7 | ActiveSupport.on_load(:action_controller) do 8 | wrap_parameters format: [:json] 9 | end 10 | 11 | # To enable root element in JSON for ActiveRecord objects. 12 | # ActiveSupport.on_load(:active_record) do 13 | # self.include_root_in_json = true 14 | # end 15 | -------------------------------------------------------------------------------- /bin/spring: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env ruby 2 | 3 | # This file loads Spring without using Bundler, in order to be fast. 4 | # It gets overwritten when you run the `spring binstub` command. 5 | 6 | unless defined?(Spring) 7 | require 'rubygems' 8 | require 'bundler' 9 | 10 | lockfile = Bundler::LockfileParser.new(Bundler.default_lockfile.read) 11 | spring = lockfile.specs.detect { |spec| spec.name == 'spring' } 12 | if spring 13 | Gem.use_paths Gem.dir, Bundler.bundle_path.to_s, *Gem.path 14 | gem 'spring', spring.version 15 | require 'spring/binstub' 16 | end 17 | end 18 | -------------------------------------------------------------------------------- /app/views/users/passwords/new.html.erb: -------------------------------------------------------------------------------- 1 |

Forgot your password?

2 | 3 | <%= form_for(resource, as: resource_name, url: password_path(resource_name), html: { method: :post }) do |f| %> 4 | <%= render "devise/shared/error_messages", resource: resource %> 5 | 6 |
7 | <%= f.label :email %>
8 | <%= f.email_field :email, autofocus: true, autocomplete: "email" %> 9 |
10 | 11 |
12 | <%= f.submit "Send me reset password instructions" %> 13 |
14 | <% end %> 15 | 16 | <%= render "users/shared/links" %> 17 | -------------------------------------------------------------------------------- /db/seeds.rb: -------------------------------------------------------------------------------- 1 | # This file should contain all the record creation needed to seed the database with its default values. 2 | # The data can then be loaded with the rails db:seed command (or created alongside the database with db:setup). 3 | # 4 | # Examples: 5 | # 6 | # movies = Movie.create([{ name: 'Star Wars' }, { name: 'Lord of the Rings' }]) 7 | # Character.create(name: 'Luke', movie: movies.first) 8 | ENV["USERS"]&.split(",")&.each{|user| User.find_by(email: "#{user}@qz.com") or User.create!({email: "#{user}@qz.com", password: ENV["DASHBOARDPW"], password_confirmation: ENV["DASHBOARDPW"]}) } 9 | -------------------------------------------------------------------------------- /app/views/users/confirmations/new.html.erb: -------------------------------------------------------------------------------- 1 |

Resend confirmation instructions

2 | 3 | <%= form_for(resource, as: resource_name, url: confirmation_path(resource_name), html: { method: :post }) do |f| %> 4 | <%= render "devise/shared/error_messages", resource: resource %> 5 | 6 |
7 | <%= f.label :email %>
8 | <%= f.email_field :email, autofocus: true, autocomplete: "email", value: (resource.pending_reconfirmation? ? resource.unconfirmed_email : resource.email) %> 9 |
10 | 11 |
12 | <%= f.submit "Resend confirmation instructions" %> 13 |
14 | <% end %> 15 | 16 | <%= render "users/shared/links" %> 17 | -------------------------------------------------------------------------------- /app/controllers/users/sessions_controller.rb: -------------------------------------------------------------------------------- 1 | # frozen_string_literal: true 2 | 3 | class Users::SessionsController < Devise::SessionsController 4 | # before_action :configure_sign_in_params, only: [:create] 5 | 6 | # GET /resource/sign_in 7 | # def new 8 | # super 9 | # end 10 | 11 | # POST /resource/sign_in 12 | # def create 13 | # super 14 | # end 15 | 16 | # DELETE /resource/sign_out 17 | # def destroy 18 | # super 19 | # end 20 | 21 | # protected 22 | 23 | # If you have extra params to permit, append them to the sanitizer. 24 | # def configure_sign_in_params 25 | # devise_parameter_sanitizer.permit(:sign_in, keys: [:attribute]) 26 | # end 27 | end 28 | -------------------------------------------------------------------------------- /lib/tasks/jobs.rake: -------------------------------------------------------------------------------- 1 | namespace :jobs do 2 | task create: :environment do 3 | [ 4 | ["ad_archive_report:daily", 24, 1], 5 | ["text:fbpac_ads", 1, 1], 6 | ["text:ads", 6, 1], 7 | ["denormalize:payers", 1, 1], 8 | ["swing_states:get", 24, 1], 9 | ["text:topics", 6, 1], 10 | ["pac-archiver", 24, 1], 11 | ["fbpac-classifier", 1, 1], 12 | ["fbpac-waist-parser", 1, 1], 13 | ].each do |job_name, expected_run_rate, expected_duration| 14 | job = Job.find_or_create_by(name: job_name) 15 | job.expected_run_rate = expected_run_rate 16 | job.estimated_duration = expected_duration 17 | job.save 18 | end 19 | end 20 | end 21 | -------------------------------------------------------------------------------- /config/initializers/assets.rb: -------------------------------------------------------------------------------- 1 | # Be sure to restart your server when you modify this file. 2 | 3 | # Version of your assets, change this if you want to expire all your assets. 4 | Rails.application.config.assets.version = '1.0' 5 | 6 | # Add additional assets to the asset load path. 7 | # Rails.application.config.assets.paths << Emoji.images_path 8 | # Add Yarn node_modules folder to the asset load path. 9 | Rails.application.config.assets.paths << Rails.root.join('node_modules') 10 | 11 | # Precompile additional assets. 12 | # application.js, application.css, and all non-JS/CSS in the app/assets 13 | # folder are already added. 14 | # Rails.application.config.assets.precompile += %w( admin.js admin.css ) 15 | -------------------------------------------------------------------------------- /app/views/layouts/application.html.erb: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | AtiDashboard 5 | <%= csrf_meta_tags %> 6 | <%= csp_meta_tag %> 7 | 8 | <%= stylesheet_link_tag 'application', media: 'all', 'data-turbolinks-track': 'reload' %> 9 | <%= javascript_pack_tag 'application', 'data-turbolinks-track': 'reload' %> 10 | 11 | 12 | 13 |
14 | <% if notice %> 15 |

<%= notice %>

16 | <% end %> 17 | <% if alert %> 18 |

<%= alert %>

19 | <% end %> 20 | 21 | 22 | <%= yield %> 23 |
24 | 25 | 26 | -------------------------------------------------------------------------------- /config/initializers/inflections.rb: -------------------------------------------------------------------------------- 1 | # Be sure to restart your server when you modify this file. 2 | 3 | # Add new inflection rules using the following format. Inflections 4 | # are locale specific, and you may define rules for as many different 5 | # locales as you wish. All of these examples are active by default: 6 | # ActiveSupport::Inflector.inflections(:en) do |inflect| 7 | # inflect.plural /^(ox)$/i, '\1en' 8 | # inflect.singular /^(ox)en/i, '\1' 9 | # inflect.irregular 'person', 'people' 10 | # inflect.uncountable %w( fish sheep ) 11 | # end 12 | 13 | # These inflection rules are supported but not enabled by default: 14 | # ActiveSupport::Inflector.inflections(:en) do |inflect| 15 | # inflect.acronym 'RESTful' 16 | # end 17 | -------------------------------------------------------------------------------- /app/controllers/users/unlocks_controller.rb: -------------------------------------------------------------------------------- 1 | # frozen_string_literal: true 2 | 3 | class Users::UnlocksController < Devise::UnlocksController 4 | # GET /resource/unlock/new 5 | # def new 6 | # super 7 | # end 8 | 9 | # POST /resource/unlock 10 | # def create 11 | # super 12 | # end 13 | 14 | # GET /resource/unlock?unlock_token=abcdef 15 | # def show 16 | # super 17 | # end 18 | 19 | # protected 20 | 21 | # The path used after sending unlock password instructions 22 | # def after_sending_unlock_instructions_path_for(resource) 23 | # super(resource) 24 | # end 25 | 26 | # The path used after unlocking the resource 27 | # def after_unlock_path_for(resource) 28 | # super(resource) 29 | # end 30 | end 31 | -------------------------------------------------------------------------------- /app/models/ad_archive_report_page.rb: -------------------------------------------------------------------------------- 1 | class AdArchiveReportPage < ApplicationRecord 2 | belongs_to :page, foreign_key: :page_id, primary_key: :page_id, optional: true 3 | belongs_to :writable_page, foreign_key: :page_id, primary_key: :page_id, optional: true 4 | belongs_to :ad_archive_report 5 | default_scope { order(:ad_archive_report_id) } 6 | 7 | def has_disclaimer? 8 | disclaimer != "These ads ran without a disclaimer" 9 | end 10 | 11 | def ad_library_url 12 | url = "https://www.facebook.com/ads/library/?active_status=all&ad_type=political_and_issue_ads&country=US&impression_search_field=has_impressions_lifetime&q=#{page_name}&page_ids[0]=#{page_id}" 13 | url += "&disclaimer_texts[0]=#{disclaimer}" if has_disclaimer? 14 | url 15 | end 16 | end 17 | -------------------------------------------------------------------------------- /app/views/users/sessions/new.html.erb: -------------------------------------------------------------------------------- 1 |

Log in

2 | 3 | <%= form_for(resource, as: resource_name, url: session_path(resource_name)) do |f| %> 4 |
5 | <%= f.label :email %>
6 | <%= f.email_field :email, autofocus: true, autocomplete: "email" %> 7 |
8 | 9 |
10 | <%= f.label :password %>
11 | <%= f.password_field :password, autocomplete: "current-password" %> 12 |
13 | 14 | <% if devise_mapping.rememberable? %> 15 |
16 | <%= f.check_box :remember_me %> 17 | <%= f.label :remember_me %> 18 |
19 | <% end %> 20 | 21 |
22 | <%= f.submit "Log in" %> 23 |
24 | <% end %> 25 | 26 | <%= render "users/shared/links" %> 27 | -------------------------------------------------------------------------------- /config/database.yml: -------------------------------------------------------------------------------- 1 | # SQLite. Versions 3.8.0 and up are supported. 2 | # gem install sqlite3 3 | # 4 | # Ensure the SQLite 3 gem is defined in your Gemfile 5 | # gem 'sqlite3' 6 | # 7 | default: &default 8 | adapter: postgresql 9 | pool: 5 10 | timeout: 5000 11 | 12 | development: 13 | <<: *default 14 | database: fbati 15 | 16 | # Warning: The database defined as "test" will be erased and 17 | # re-generated from your development database when you run "rake". 18 | # Do not set this db to the same as development or production. 19 | test: 20 | adapter: sqlite3 21 | database: db/test.sqlite3 22 | 23 | production: 24 | <<: *default 25 | database: qzaifbdb 26 | host: <%= ENV["PGHOST"] %> 27 | password: <%= ENV["PGPASSWORD"] %> 28 | username: <%= ENV["PGUSER"] %> 29 | port: <%= ENV["PGPORT"] %> 30 | -------------------------------------------------------------------------------- /app/controllers/users/confirmations_controller.rb: -------------------------------------------------------------------------------- 1 | # frozen_string_literal: true 2 | 3 | class Users::ConfirmationsController < Devise::ConfirmationsController 4 | # GET /resource/confirmation/new 5 | # def new 6 | # super 7 | # end 8 | 9 | # POST /resource/confirmation 10 | # def create 11 | # super 12 | # end 13 | 14 | # GET /resource/confirmation?confirmation_token=abcdef 15 | # def show 16 | # super 17 | # end 18 | 19 | # protected 20 | 21 | # The path used after resending confirmation instructions. 22 | # def after_resending_confirmation_instructions_path_for(resource_name) 23 | # super(resource_name) 24 | # end 25 | 26 | # The path used after confirmation. 27 | # def after_confirmation_path_for(resource_name, resource) 28 | # super(resource_name, resource) 29 | # end 30 | end 31 | -------------------------------------------------------------------------------- /app/controllers/users/passwords_controller.rb: -------------------------------------------------------------------------------- 1 | # frozen_string_literal: true 2 | 3 | class Users::PasswordsController < Devise::PasswordsController 4 | # GET /resource/password/new 5 | # def new 6 | # super 7 | # end 8 | 9 | # POST /resource/password 10 | # def create 11 | # super 12 | # end 13 | 14 | # GET /resource/password/edit?reset_password_token=abcdef 15 | # def edit 16 | # super 17 | # end 18 | 19 | # PUT /resource/password 20 | # def update 21 | # super 22 | # end 23 | 24 | # protected 25 | 26 | # def after_resetting_password_path_for(resource) 27 | # super(resource) 28 | # end 29 | 30 | # The path used after sending reset password instructions 31 | # def after_sending_reset_password_instructions_path_for(resource_name) 32 | # super(resource_name) 33 | # end 34 | end 35 | -------------------------------------------------------------------------------- /app/controllers/users/omniauth_callbacks_controller.rb: -------------------------------------------------------------------------------- 1 | # frozen_string_literal: true 2 | 3 | class Users::OmniauthCallbacksController < Devise::OmniauthCallbacksController 4 | # You should configure your model like this: 5 | # devise :omniauthable, omniauth_providers: [:twitter] 6 | 7 | # You should also create an action method in this controller like this: 8 | # def twitter 9 | # end 10 | 11 | # More info at: 12 | # https://github.com/plataformatec/devise#omniauth 13 | 14 | # GET|POST /resource/auth/twitter 15 | # def passthru 16 | # super 17 | # end 18 | 19 | # GET|POST /users/auth/twitter/callback 20 | # def failure 21 | # super 22 | # end 23 | 24 | # protected 25 | 26 | # The path used when OmniAuth fails 27 | # def after_omniauth_failure_path_for(scope) 28 | # super(scope) 29 | # end 30 | end 31 | -------------------------------------------------------------------------------- /appspec.yml: -------------------------------------------------------------------------------- 1 | version: 0.0 2 | os: linux 3 | files: 4 | - source: / 5 | destination: /home/ubuntu/rails 6 | # for help, consult https://blog.shikisoft.com/automate-ruby-on-rails-deployments-aws-codedeploy/ 7 | permissions: 8 | - object: /home/ubuntu/rails 9 | owner: ubuntu 10 | group: ubuntu 11 | mode: 755 12 | pattern: "*" 13 | type: 14 | - file 15 | - directory 16 | hooks: 17 | BeforeInstall: 18 | - location: deploy/pre-deploy.sh 19 | timeout: 3600 20 | runas: root 21 | AfterInstall: 22 | - location: deploy/install_dependencies.sh 23 | timeout: 3600 24 | runas: ubuntu 25 | ApplicationStart: 26 | - location: deploy/start_server.sh 27 | timeout: 300 28 | runas: root 29 | ApplicationStop: 30 | - location: deploy/stop_server.sh 31 | timeout: 300 32 | runas: root -------------------------------------------------------------------------------- /lib/tasks/interim.rake: -------------------------------------------------------------------------------- 1 | 2 | require 'csv' 3 | namespace :fake do 4 | task topics: :environment do 5 | topics = ["immigrationfake", "budgetfake", "corruptionfake", "abortionfake" ] 6 | topics.each do |topic| 7 | t = Topic.find_or_create_by(topic: topic) 8 | end 9 | Ad.all.each do |ad| 10 | unless ad.writable_ad 11 | wa = WritableAd.new 12 | ad.writable_ad = wa 13 | wa.save 14 | end 15 | ad.topics << Topic.find_by(topic: topics.sample) 16 | end 17 | end 18 | end 19 | 20 | namespace :interim do 21 | task ad_ids: :environment do 22 | CSV.open("/Users/jmerrill/code/impeachmentads/impeachment_ad_ids.csv").each do |row| 23 | ad_archive_id = row[0] 24 | ad_id = row[1] 25 | ad = Ad.find_by(archive_id: ad_archive_id) 26 | next unless ad 27 | ad.ad_id = ad_id.to_i 28 | ad.save!(validate: false) 29 | end 30 | 31 | end 32 | 33 | end -------------------------------------------------------------------------------- /app/assets/stylesheets/application.css: -------------------------------------------------------------------------------- 1 | /* 2 | * This is a manifest file that'll be compiled into application.css, which will include all the files 3 | * listed below. 4 | * 5 | * Any CSS and SCSS file within this directory, lib/assets/stylesheets, or any plugin's 6 | * vendor/assets/stylesheets directory can be referenced here using a relative path. 7 | * 8 | * You're free to add application-wide styles to this file and they'll appear at the bottom of the 9 | * compiled file so the styles you add here take precedence over styles defined in any other CSS/SCSS 10 | * files in this directory. Styles in this file should be added after the last require_* statement. 11 | * It is generally better to create a new file per style scope. 12 | * 13 | *= require_tree . 14 | *= require_self 15 | */ 16 | 17 | #main { 18 | width: 1024px; 19 | margin-left: auto; 20 | margin-right: auto; 21 | } -------------------------------------------------------------------------------- /lib/tasks/page_ids.rake: -------------------------------------------------------------------------------- 1 | require 'nokogiri' 2 | 3 | namespace :page_ids do 4 | 5 | task fbpac_ads: :environment do 6 | counter = 0 7 | 8 | batch_size = 500 9 | FbpacAd.where("created_at > '2020-01-01'").where("page_id is null and advertiser is not null").find_in_batches(batch_size: batch_size) do |ads| 10 | ads.each do |ad| 11 | next if ad.html.include?("ego_unit") 12 | match = ad.html.match(/data-hovercard="https:\/\/www.facebook.com\/(\d+)"/) 13 | next unless match 14 | ad.page_id = match[1].to_i 15 | ad.save 16 | end 17 | end 18 | 19 | RestClient.post( 20 | ENV["SLACKWH"], 21 | JSON.dump({"text" => "page ID parsing for collector ads went swimmingly. (#{counter} batches processed)" }), 22 | {:content_type => "application/json"} 23 | ) if counter > 0 && ENV["SLACKWH"] 24 | end 25 | end 26 | -------------------------------------------------------------------------------- /db/migrate/20200205171023_add_search_text_to_ad_texts.rb: -------------------------------------------------------------------------------- 1 | class AddSearchTextToAdTexts < ActiveRecord::Migration[6.0] 2 | def up 3 | add_column :ad_texts, :search_text, :text 4 | add_column :ad_texts, :tsv, :tsvector 5 | add_index :ad_texts, :tsv, using: :gin 6 | 7 | execute <<-SQL 8 | CREATE TRIGGER tsvectorupdate BEFORE INSERT OR UPDATE 9 | ON ad_texts FOR EACH ROW EXECUTE PROCEDURE 10 | tsvector_update_trigger( 11 | tsv, 'pg_catalog.english', search_text 12 | ); 13 | SQL 14 | 15 | now = Time.current.to_s(:db) 16 | update("UPDATE ad_texts SET updated_at = '#{now}'") 17 | end 18 | 19 | def down 20 | execute <<-SQL 21 | DROP TRIGGER tsvectorupdate 22 | ON ad_texts 23 | SQL 24 | 25 | remove_index :ad_texts, :tsv 26 | remove_column :ad_texts, :tsv 27 | remove_column :ad_texts, :search_text 28 | end 29 | 30 | 31 | end 32 | -------------------------------------------------------------------------------- /app/views/users/passwords/edit.html.erb: -------------------------------------------------------------------------------- 1 |

Change your password

2 | 3 | <%= form_for(resource, as: resource_name, url: password_path(resource_name), html: { method: :put }) do |f| %> 4 | <%= render "devise/shared/error_messages", resource: resource %> 5 | <%= f.hidden_field :reset_password_token %> 6 | 7 |
8 | <%= f.label :password, "New password" %>
9 | <% if @minimum_password_length %> 10 | (<%= @minimum_password_length %> characters minimum)
11 | <% end %> 12 | <%= f.password_field :password, autofocus: true, autocomplete: "new-password" %> 13 |
14 | 15 |
16 | <%= f.label :password_confirmation, "Confirm new password" %>
17 | <%= f.password_field :password_confirmation, autocomplete: "new-password" %> 18 |
19 | 20 |
21 | <%= f.submit "Change my password" %> 22 |
23 | <% end %> 24 | 25 | <%= render "users/shared/links" %> 26 | -------------------------------------------------------------------------------- /app/views/users/registrations/new.html.erb: -------------------------------------------------------------------------------- 1 |

Sign up

2 | 3 | <%= form_for(resource, as: resource_name, url: registration_path(resource_name)) do |f| %> 4 | <%= render "devise/shared/error_messages", resource: resource %> 5 | 6 |
7 | <%= f.label :email %>
8 | <%= f.email_field :email, autofocus: true, autocomplete: "email" %> 9 |
10 | 11 |
12 | <%= f.label :password %> 13 | <% if @minimum_password_length %> 14 | (<%= @minimum_password_length %> characters minimum) 15 | <% end %>
16 | <%= f.password_field :password, autocomplete: "new-password" %> 17 |
18 | 19 |
20 | <%= f.label :password_confirmation %>
21 | <%= f.password_field :password_confirmation, autocomplete: "new-password" %> 22 |
23 | 24 |
25 | <%= f.submit "Sign up" %> 26 |
27 | <% end %> 28 | 29 | <%= render "users/shared/links" %> 30 | -------------------------------------------------------------------------------- /app/views/users/registrations/new_other.html.erb: -------------------------------------------------------------------------------- 1 |

Sign up

2 | 3 | <%= form_for(resource, as: resource_name, url: other_users_create_path(resource_name)) do |f| %> 4 | <%= render "devise/shared/error_messages", resource: resource %> 5 | 6 |
7 | <%= f.label :email %>
8 | <%= f.email_field :email, autofocus: true, autocomplete: "email" %> 9 |
10 | 11 |
12 | <%= f.label :password %> 13 | <% if @minimum_password_length %> 14 | (<%= @minimum_password_length %> characters minimum) 15 | <% end %>
16 | <%= f.password_field :password, autocomplete: "new-password" %> 17 |
18 | 19 |
20 | <%= f.label :password_confirmation %>
21 | <%= f.password_field :password_confirmation, autocomplete: "new-password" %> 22 |
23 | 24 |
25 | <%= f.submit "Sign up" %> 26 |
27 | <% end %> 28 | 29 | <%= render "users/shared/links" %> 30 | -------------------------------------------------------------------------------- /config/locales/en.yml: -------------------------------------------------------------------------------- 1 | # Files in the config/locales directory are used for internationalization 2 | # and are automatically loaded by Rails. If you want to use locales other 3 | # than English, add the necessary files in this directory. 4 | # 5 | # To use the locales, use `I18n.t`: 6 | # 7 | # I18n.t 'hello' 8 | # 9 | # In views, this is aliased to just `t`: 10 | # 11 | # <%= t('hello') %> 12 | # 13 | # To use a different locale, set it with `I18n.locale`: 14 | # 15 | # I18n.locale = :es 16 | # 17 | # This would use the information in config/locales/es.yml. 18 | # 19 | # The following keys must be escaped otherwise they will not be retrieved by 20 | # the default I18n backend: 21 | # 22 | # true, false, on, off, yes, no 23 | # 24 | # Instead, surround them with single quotes. 25 | # 26 | # en: 27 | # 'true': 'foo' 28 | # 29 | # To learn more, please read the Rails Internationalization guide 30 | # available at https://guides.rubyonrails.org/i18n.html. 31 | 32 | en: 33 | hello: "Hello world" 34 | -------------------------------------------------------------------------------- /lib/tasks/core_advertisers.rake: -------------------------------------------------------------------------------- 1 | require 'csv' 2 | 3 | namespace :core_advertisers do 4 | task refresh: :environment do 5 | GOP_OTHERS = [706716899745696, 607626319739286, 1771156219840594, 612701059241880].map{|id| [id, nil, nil, nil, 'gop']} # team trump, women for trump, black voices for trump, latinos for trump 6 | 7 | page_ids = CSV.open("core_advertisers_20200422.csv").select{ |row| row[3] == "TRUE" } + GOP_OTHERS 8 | page_ids.each do |row| 9 | wpage = WritablePage.find_or_create_by(page_id: row[0].to_s.gsub("'", '')) 10 | wpage.core = true 11 | wpage.partisanship = row[4] 12 | wpage.save 13 | end 14 | 15 | noncore_page_ids = CSV.open("core_advertisers_20200422.csv").select{ |row| row[3] == "FALSE" } 16 | noncore_page_ids.each do |row| 17 | wpage = WritablePage.find_or_create_by(page_id: row[0].to_s.gsub("'", '')) 18 | wpage.core = false 19 | wpage.partisanship = nil 20 | wpage.save 21 | end 22 | end 23 | end 24 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright 2020 Quartz 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: 4 | 5 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 6 | 7 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 8 | 9 | 10 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # See https://help.github.com/articles/ignoring-files for more about ignoring files. 2 | # 3 | # If you find yourself ignoring temporary files generated by your text editor 4 | # or operating system, you probably want to add a global ignore instead: 5 | # git config --global core.excludesfile '~/.gitignore_global' 6 | 7 | # Ignore bundler config. 8 | /.bundle 9 | 10 | # Ignore the default SQLite database. 11 | /db/*.sqlite3 12 | /db/*.sqlite3-journal 13 | 14 | 15 | /db/structure.sql 16 | 17 | # Ignore all logfiles and tempfiles. 18 | /log/* 19 | /tmp/* 20 | !/log/.keep 21 | !/tmp/.keep 22 | 23 | test/ 24 | 25 | # Ignore uploaded files in development. 26 | /storage/* 27 | !/storage/.keep 28 | 29 | /public/assets 30 | .byebug_history 31 | 32 | # Ignore master key for decrypting credentials and more. 33 | /config/master.key 34 | 35 | /public/packs 36 | /public/packs-test 37 | /node_modules 38 | /yarn-error.log 39 | yarn-debug.log* 40 | .yarn-integrity 41 | 42 | secrets.env 43 | terraform.tf 44 | .terraform/ 45 | deploy/ 46 | terraform.tfstate 47 | terraform.tfstate.backup 48 | -------------------------------------------------------------------------------- /app/models/ad_archive_report.rb: -------------------------------------------------------------------------------- 1 | class AdArchiveReport < ApplicationRecord 2 | default_scope{ order :scrape_date } 3 | has_many :ad_archive_report_pages 4 | 5 | 6 | # this is the date against which we'll do total calculations in the dashboard 7 | # since we don't want to use May 2018 as the date against which totals are calculated forever 8 | # (since that'll include all sorts of irrelevant stuff, like Beto for Texas) 9 | START_DATE = Date.new(2019, 11, 17) # after LAGov elex. 10 | 11 | 12 | def self.about_a_week_ago 13 | most_recent = AdArchiveReport.where(kind: "lifelong").last.scrape_date.to_date.to_s 14 | AdArchiveReport.unscope(:order).where(kind: "lifelong").where("scrape_date < ?", most_recent).order("abs(7 - extract(day from '#{most_recent}' - scrape_date))").first 15 | end 16 | 17 | def self.starting_point 18 | AdArchiveReport.unscope(:order).where(kind: "lifelong", loaded: true).where("scrape_date >= ?", START_DATE).order("scrape_date asc").first 19 | end 20 | 21 | def filename 22 | # pick a /tmp URL 23 | # TODO: download it from s3_url if the /tmp file doesn't exist (or is empty) 24 | # return the tmp url 25 | s3_url 26 | end 27 | 28 | end 29 | -------------------------------------------------------------------------------- /bin/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env ruby 2 | require 'fileutils' 3 | 4 | # path to your application root. 5 | APP_ROOT = File.expand_path('..', __dir__) 6 | 7 | def system!(*args) 8 | system(*args) || abort("\n== Command #{args} failed ==") 9 | end 10 | 11 | FileUtils.chdir APP_ROOT do 12 | # This script is a way to setup or update your development environment automatically. 13 | # This script is idempotent, so that you can run it at anytime and get an expectable outcome. 14 | # Add necessary setup steps to this file. 15 | 16 | puts '== Installing dependencies ==' 17 | system! 'gem install bundler --conservative' 18 | system('bundle check') || system!('bundle install') 19 | 20 | # Install JavaScript dependencies 21 | # system('bin/yarn') 22 | 23 | # puts "\n== Copying sample files ==" 24 | # unless File.exist?('config/database.yml') 25 | # FileUtils.cp 'config/database.yml.sample', 'config/database.yml' 26 | # end 27 | 28 | puts "\n== Preparing database ==" 29 | system! 'bin/rails db:prepare' 30 | 31 | puts "\n== Removing old logs and tempfiles ==" 32 | system! 'bin/rails log:clear tmp:clear' 33 | 34 | puts "\n== Restarting application server ==" 35 | system! 'bin/rails restart' 36 | end 37 | -------------------------------------------------------------------------------- /config/storage.yml: -------------------------------------------------------------------------------- 1 | test: 2 | service: Disk 3 | root: <%= Rails.root.join("tmp/storage") %> 4 | 5 | local: 6 | service: Disk 7 | root: <%= Rails.root.join("storage") %> 8 | 9 | # Use rails credentials:edit to set the AWS secrets (as aws:access_key_id|secret_access_key) 10 | # amazon: 11 | # service: S3 12 | # access_key_id: <%= Rails.application.credentials.dig(:aws, :access_key_id) %> 13 | # secret_access_key: <%= Rails.application.credentials.dig(:aws, :secret_access_key) %> 14 | # region: us-east-1 15 | # bucket: your_own_bucket 16 | 17 | # Remember not to checkin your GCS keyfile to a repository 18 | # google: 19 | # service: GCS 20 | # project: your_project 21 | # credentials: <%= Rails.root.join("path/to/gcs.keyfile") %> 22 | # bucket: your_own_bucket 23 | 24 | # Use rails credentials:edit to set the Azure Storage secret (as azure_storage:storage_access_key) 25 | # microsoft: 26 | # service: AzureStorage 27 | # storage_account_name: your_account_name 28 | # storage_access_key: <%= Rails.application.credentials.dig(:azure_storage, :storage_access_key) %> 29 | # container: your_container_name 30 | 31 | # mirror: 32 | # service: Mirror 33 | # primary: local 34 | # mirrors: [ amazon, google, microsoft ] 35 | -------------------------------------------------------------------------------- /app/views/users/shared/_links.html.erb: -------------------------------------------------------------------------------- 1 | <%- if controller_name != 'sessions' %> 2 | <%= link_to "Log in", new_session_path(resource_name) %>
3 | <% end %> 4 | 5 | <%- if devise_mapping.registerable? && controller_name != 'registrations' %> 6 | <%= link_to "Sign up", new_registration_path(resource_name) %>
7 | <% end %> 8 | 9 | <%- if devise_mapping.recoverable? && controller_name != 'passwords' && controller_name != 'registrations' %> 10 | <%= link_to "Forgot your password?", new_password_path(resource_name) %>
11 | <% end %> 12 | 13 | <%- if devise_mapping.confirmable? && controller_name != 'confirmations' %> 14 | <%= link_to "Didn't receive confirmation instructions?", new_confirmation_path(resource_name) %>
15 | <% end %> 16 | 17 | <%- if devise_mapping.lockable? && resource_class.unlock_strategy_enabled?(:email) && controller_name != 'unlocks' %> 18 | <%= link_to "Didn't receive unlock instructions?", new_unlock_path(resource_name) %>
19 | <% end %> 20 | 21 | <%- if devise_mapping.omniauthable? %> 22 | <%- resource_class.omniauth_providers.each do |provider| %> 23 | <%= link_to "Sign in with #{OmniAuth::Utils.camelize(provider)}", omniauth_authorize_path(resource_name, provider) %>
24 | <% end %> 25 | <% end %> 26 | -------------------------------------------------------------------------------- /app/views/fbpac_ads/pivot.html.erb: -------------------------------------------------------------------------------- 1 |

Quartz Political Facebook Ads Dashboard

2 | 3 |

top <%= @kind_of_thing %> <%= @first_seen ? "FIRST seen" : "seen" %> in the past <%= @time_count %> <%= @time_unit %>

4 | 5 |
6 | 7 | 8 | <% @pivot.each do |thing, cnt| %> 9 | 10 | 27 | 28 | 29 | <% end %> 30 | 31 |
11 | <% if @kind_of_thing == "segments" %> 12 | 13 | <% elsif @kind_of_thing == "targets" %> 14 | 15 | <% elsif @kind_of_thing == "paid_for_by" %> 16 | <% if thing.nil? %> 17 | 18 | <% else %> 19 | 20 | <% end %> 21 | <% elsif @kind_of_thing == "advertiser" %> 22 | 23 | <% end %> 24 | <%= @kind_of_thing == "segments" ? thing.compact.join(" → ") : (thing.nil? ? '(none)' : thing) %> 25 | 26 | <%= cnt %>
32 |
-------------------------------------------------------------------------------- /app/models/writable_ad.rb: -------------------------------------------------------------------------------- 1 | 2 | require 'aws-sdk-s3' 3 | 4 | class WritableAd < ApplicationRecord 5 | belongs_to :ad, primary_key: :archive_id, foreign_key: :archive_id, optional: true 6 | belongs_to :ad_text, primary_key: :text_hash, foreign_key: :text_hash, optional: true 7 | belongs_to :fbpac_ad, primary_key: :id, foreign_key: :ad_id, optional: true 8 | # belongs_to :collector_ad 9 | belongs_to :page, primary_key: :page_id, foreign_key: :page_id, optional: true 10 | belongs_to :writable_page, primary_key: :page_id, foreign_key: :page_id, optional: true 11 | 12 | 13 | # for screenshots derived from Laura's DB. 14 | BUCKET_NAME = "qz-aistudio-fbpac-ads" 15 | def gcs_url 16 | ENV["GCS_URL"] + archive_id.to_s + ".png" 17 | end 18 | def generate_s3_url 19 | "s3:///#{BUCKET_NAME}/#{s3_path}" 20 | end 21 | def s3_path 22 | "screenshots/#{archive_id}.png" 23 | end 24 | 25 | def http_s3_url 26 | "https://qz-aistudio-fbpac-ads.s3.us-east-2.amazonaws.com/#{s3_path}" 27 | end 28 | 29 | def copy_screenshot_to_s3! 30 | return if s3_url 31 | s3 = Aws::S3::Resource.new(region:'us-east-2') 32 | obj = s3.bucket(BUCKET_NAME).object(s3_path) 33 | unless obj.exists? 34 | begin 35 | img_data = RestClient.get(gcs_url) 36 | rescue 37 | return 38 | end 39 | obj.put(body: img_data.body, acl: "public-read") 40 | end 41 | self.s3_url = generate_s3_url 42 | self.save 43 | end 44 | 45 | 46 | end 47 | -------------------------------------------------------------------------------- /config/initializers/content_security_policy.rb: -------------------------------------------------------------------------------- 1 | # Be sure to restart your server when you modify this file. 2 | 3 | # Define an application-wide content security policy 4 | # For further information see the following documentation 5 | # https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy 6 | 7 | # Rails.application.config.content_security_policy do |policy| 8 | # policy.default_src :self, :https 9 | # policy.font_src :self, :https, :data 10 | # policy.img_src :self, :https, :data 11 | # policy.object_src :none 12 | # policy.script_src :self, :https 13 | # policy.style_src :self, :https 14 | # # If you are using webpack-dev-server then specify webpack-dev-server host 15 | # policy.connect_src :self, :https, "http://localhost:3035", "ws://localhost:3035" if Rails.env.development? 16 | 17 | # # Specify URI for violation reports 18 | # # policy.report_uri "/csp-violation-report-endpoint" 19 | # end 20 | 21 | # If you are using UJS then enable automatic nonce generation 22 | # Rails.application.config.content_security_policy_nonce_generator = -> request { SecureRandom.base64(16) } 23 | 24 | # Set the nonce only to specific directives 25 | # Rails.application.config.content_security_policy_nonce_directives = %w(script-src) 26 | 27 | # Report CSP violations to a specified URI 28 | # For further information see the following documentation: 29 | # https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy-Report-Only 30 | # Rails.application.config.content_security_policy_report_only = true 31 | -------------------------------------------------------------------------------- /app/views/pages/bigspenders.html.erb: -------------------------------------------------------------------------------- 1 | 10 |

Big Spenders

11 | 12 |

pages that have spent a lot of money between <%= AdArchiveReport.find(@big_spenders.first.previous_ad_archive_report_id).scrape_date %> and <%= @big_spenders.first.ad_archive_report.scrape_date %>.

13 | 14 |

How to use this page: Check out these advertisers and see if any of them look weird.

15 | 16 | <% @big_spenders.each_with_index do |big_spender, idx| %> 17 | 18 |
19 |

<%= big_spender.is_new ? "🚨NEW! " : '' %><%= big_spender.ad_archive_report_page.page_name %>

(overview)
20 |

Paid for by <%= big_spender.ad_archive_report_page.disclaimer %>

(overview)
21 |

$<%= big_spender.spend_amount.to_s.reverse.scan(/\d{3}|.+/).join(",").reverse %> spent in <%= big_spender.duration_days %> days (Ads on FB)

22 | 23 | <%= text_area_tag idx.to_s + "-notes", big_spender.writable_page&.notes, cols: 60, rows: 6, class: "onkeyupdelay", data: {page_id: big_spender.page_id, disclaimer: big_spender.ad_archive_report_page.disclaimer} %> 24 |
25 | <% end %> 26 | 27 | 28 | -------------------------------------------------------------------------------- /db/migrate/20191106220234_devise_create_users.rb: -------------------------------------------------------------------------------- 1 | # frozen_string_literal: true 2 | 3 | class DeviseCreateUsers < ActiveRecord::Migration[6.0] 4 | def change 5 | create_table :users do |t| 6 | ## Database authenticatable 7 | t.string :email, null: false, default: "" 8 | t.string :encrypted_password, null: false, default: "" 9 | 10 | ## Recoverable 11 | t.string :reset_password_token 12 | t.datetime :reset_password_sent_at 13 | 14 | ## Rememberable 15 | t.datetime :remember_created_at 16 | 17 | ## Trackable 18 | # t.integer :sign_in_count, default: 0, null: false 19 | # t.datetime :current_sign_in_at 20 | # t.datetime :last_sign_in_at 21 | # t.inet :current_sign_in_ip 22 | # t.inet :last_sign_in_ip 23 | 24 | ## Confirmable 25 | # t.string :confirmation_token 26 | # t.datetime :confirmed_at 27 | # t.datetime :confirmation_sent_at 28 | # t.string :unconfirmed_email # Only if using reconfirmable 29 | 30 | ## Lockable 31 | # t.integer :failed_attempts, default: 0, null: false # Only if lock strategy is :failed_attempts 32 | # t.string :unlock_token # Only if unlock strategy is :email or :both 33 | # t.datetime :locked_at 34 | 35 | 36 | t.timestamps null: false 37 | end 38 | 39 | add_index :users, :email, unique: true 40 | add_index :users, :reset_password_token, unique: true 41 | # add_index :users, :confirmation_token, unique: true 42 | # add_index :users, :unlock_token, unique: true 43 | end 44 | end 45 | -------------------------------------------------------------------------------- /config/application.rb: -------------------------------------------------------------------------------- 1 | require_relative 'boot' 2 | 3 | require 'rails/all' 4 | 5 | # Require the gems listed in Gemfile, including any gems 6 | # you've limited to :test, :development, or :production. 7 | Bundler.require(*Rails.groups) 8 | 9 | module AtiDashboard 10 | class Application < Rails::Application 11 | # Initialize configuration defaults for originally generated Rails version. 12 | config.load_defaults 6.0 13 | # config.active_record.schema_format = :sql 14 | 15 | config.middleware.insert_before 0, Rack::Cors do 16 | allow do 17 | origins 'http://localhost:3000' 18 | resource '*', headers: :any, methods: [:get], credentials: true 19 | end 20 | allow do 21 | origins 'http://pol-ad-dashboard.s3-website.us-east-2.amazonaws.com' 22 | resource '*', headers: :any, methods: [:get] 23 | end 24 | 25 | allow do 26 | origins 'https://dashboard.qz.ai' 27 | resource '*', headers: :any, methods: [:get] 28 | end 29 | allow do 30 | origins 'https://dashboard-frontend.qz.ai' 31 | resource '*', headers: :any, methods: [:get] 32 | end 33 | 34 | allow do 35 | origins 'https://projects.propublica.org' 36 | resource '/pp/*', headers: :any, methods: [:get], credentials: true 37 | end 38 | end 39 | # Settings in config/environments/* take precedence over those specified here. 40 | # Application configuration can go into files in config/initializers 41 | # -- all .rb files in that directory are automatically loaded after loading 42 | # the framework and any gems in your application. 43 | end 44 | end -------------------------------------------------------------------------------- /app/views/ads/_ad.html.erb: -------------------------------------------------------------------------------- 1 |
2 |

<%= page_name %>

3 |

Paid for by <%= ad.funding_entity %> ($<%= number_with_delimiter(wads.map{|wad| wad.ad.impressions_record.min_spend}.reduce(&:+)) %> - <%= number_with_delimiter(wads.map{|wad| wad.ad.impressions_record.max_spend}.reduce(&:+)) %> spent)

4 |

impressions: <%= number_with_delimiter(wads.map{|wad| wad.ad.impressions_record.min_impressions}.reduce(&:+)) %> - <%= number_with_delimiter(wads.map{|wad| wad.ad.impressions_record.max_impressions}.reduce(&:+)) %>

5 |

<%= ad.ad_creative_body %>

6 | <% if wad.s3_url %> 7 | <%= [page_name, ad.funding_entity, ad.ad_creative_body, ad.ad_creative_link_title, ad.ad_creative_link_description, ad.ad_creative_link_caption].join(' ') %> 8 | <% else %> 9 |

No image yet, sadly.

10 | <% end %> 11 |

see video on Facebook

12 |

<%= ad.ad_creative_link_title %>

13 |

<%= ad.ad_creative_link_description %>

14 |

<%= ad.ad_creative_link_caption %>

15 |

Swing states: <%= states.compact.join(", ") %>

16 | 17 | <% if wad.ad_text.topics.reject{|text| text == "none"}.size >= 1 %> 18 |

Topics: <%= wad.ad_text.topics.reject{|text| text == "none"}.map(&:topic) %>

19 | <% end %> 20 | <% if wad.ad_text.fbpac_ads.size >= 1 && ad.ad_creative_body %> 21 |

How was this targeted?: <%= wad.ad_text.fbpac_ads.map(&:targets).compact.first %>

22 | <% end %> 23 |
-------------------------------------------------------------------------------- /lib/tasks/topics.rake: -------------------------------------------------------------------------------- 1 | 2 | # for each ad_text without topics 3 | # throw it against the topic endpoint TOPICS_URL 4 | 5 | 6 | namespace :text do 7 | task topics: :environment do 8 | counter = 0 9 | start = Time.now 10 | WritablePage.where(core: true).each do |wpage| 11 | wpage.ad_texts.find_in_batches(batch_size: 16) do |texts| 12 | counter += texts.size 13 | retried = 0 14 | begin 15 | AdText.classify_topic(texts) 16 | rescue RestClient::BadGateway 17 | sleep 5 18 | retry if retried < 3 19 | retried += 1 20 | end 21 | puts "successful batch -- #{counter}" 22 | end 23 | end 24 | 25 | AdText.includes(:ad_topics).joins(writable_ads: [:fbpac_ad]).search_for("biden OR trump").where( :ad_topics => { :ad_text_id => nil } ).find_in_batches(batch_size: 16) do |texts| 26 | counter += texts.size 27 | retried = 0 28 | begin 29 | AdText.classify_topic(texts) 30 | rescue RestClient::BadGateway 31 | sleep 5 32 | retry if retried < 3 33 | retried += 1 34 | end 35 | puts "successful batch -- #{counter}" 36 | end 37 | 38 | job = Job.find_by(name: "text:topics") 39 | job_run = job.job_runs.create({ 40 | start_time: start, 41 | end_time: Time.now, 42 | success: true, 43 | }) 44 | 45 | 46 | RestClient.post( 47 | ENV["SLACKWH"], 48 | JSON.dump({"text" => "(6/6): Facebook ad topic classification went swimmingly. (#{counter} batches processed)" }), 49 | {:content_type => "application/json"} 50 | ) if counter > 0 51 | end 52 | end 53 | -------------------------------------------------------------------------------- /lib/tasks/payers_and_advertisers.rake: -------------------------------------------------------------------------------- 1 | namespace :denormalize do 2 | desc "create objects for Payers in the DB, denormalizing DB" 3 | task payers: :environment do 4 | start = Time.now 5 | existing_payers = Set.new(Payer.select(:name).map(&:name)) 6 | 7 | payers_created = 0 8 | Ad.unscope(:order).group(:funding_entity).count.each do |entity, cnt| 9 | next if existing_payers.include?(entity) 10 | Payer.create(name: entity) 11 | payers_created += 1 12 | end 13 | 14 | FbpacAd.unscope(:order).group(:paid_for_by).count.each do |entity, cnt| 15 | next if existing_payers.include?(entity) 16 | Payer.create(name: entity) 17 | payers_created += 1 18 | end 19 | 20 | job = Job.find_by(name: "denormalize:payers") 21 | job_run = job.job_runs.create({ 22 | start_time: start, 23 | end_time: Time.now, 24 | success: true, 25 | }) 26 | 27 | RestClient.post( 28 | ENV["SLACKWH"], 29 | JSON.dump({"text" => "(5/6): Facebook payer-denormalization went swimmingly. (#{payers_created} created)" }), 30 | {:content_type => "application/json"} 31 | ) 32 | 33 | 34 | end 35 | 36 | task advertisers: :environment do 37 | Ad.unscope(:order).group(:page_id).count.each do |page_id, cnt| 38 | WritablePage.find_or_create_by(page_id: page_id) 39 | end 40 | 41 | # TODO: what about FbpacAds? we don't have a page_id there, necessarily. 42 | 43 | RestClient.post( 44 | ENV["SLACKWH"], 45 | JSON.dump({"text" => "Facebook advertiser denormalization went swimmingly" }), 46 | {:content_type => "application/json"} 47 | ) 48 | 49 | end 50 | end 51 | 52 | -------------------------------------------------------------------------------- /app/views/youtube/index.html.erb: -------------------------------------------------------------------------------- 1 | 2 | 3 | Interim YouTube Ads Dashboard 4 | 5 | 6 | Home: FB dashboard 7 |

Jeremy's Interim YouTube Ads Dashboard

8 | 9 |
10 |

<%= @ads_count %> Ads

11 |

<%= @political_ads_count %> Political Ads

12 | 13 |
14 | 15 |
16 | Ads per political advertiser: 17 |
23 | 24 |
25 | Political targetings: 26 |
32 | 33 |
34 | Ads per advertiser: 35 |
41 | 42 |
43 | targetings: 44 |
50 | 51 | 52 | -------------------------------------------------------------------------------- /app/views/users/registrations/edit.html.erb: -------------------------------------------------------------------------------- 1 |

Edit <%= resource_name.to_s.humanize %>

2 | 3 | <%= form_for(resource, as: resource_name, url: registration_path(resource_name), html: { method: :put }) do |f| %> 4 | <%= render "devise/shared/error_messages", resource: resource %> 5 | 6 |
7 | <%= f.label :email %>
8 | <%= f.email_field :email, autofocus: true, autocomplete: "email" %> 9 |
10 | 11 | <% if devise_mapping.confirmable? && resource.pending_reconfirmation? %> 12 |
Currently waiting confirmation for: <%= resource.unconfirmed_email %>
13 | <% end %> 14 | 15 |
16 | <%= f.label :password %> (leave blank if you don't want to change it)
17 | <%= f.password_field :password, autocomplete: "new-password" %> 18 | <% if @minimum_password_length %> 19 |
20 | <%= @minimum_password_length %> characters minimum 21 | <% end %> 22 |
23 | 24 |
25 | <%= f.label :password_confirmation %>
26 | <%= f.password_field :password_confirmation, autocomplete: "new-password" %> 27 |
28 | 29 |
30 | <%= f.label :current_password %> (we need your current password to confirm your changes)
31 | <%= f.password_field :current_password, autocomplete: "current-password" %> 32 |
33 | 34 |
35 | <%= f.submit "Update" %> 36 |
37 | <% end %> 38 | 39 |

Cancel my account

40 | 41 |

Unhappy? <%= button_to "Cancel my account", registration_path(resource_name), data: { confirm: "Are you sure?" }, method: :delete %>

42 | 43 | <%= link_to "Back", :back %> 44 | -------------------------------------------------------------------------------- /config/puma.rb: -------------------------------------------------------------------------------- 1 | # Puma can serve each request in a thread from an internal thread pool. 2 | # The `threads` method setting takes two numbers: a minimum and maximum. 3 | # Any libraries that use thread pools should be configured to match 4 | # the maximum value specified for Puma. Default is set to 5 threads for minimum 5 | # and maximum; this matches the default thread size of Active Record. 6 | # 7 | max_threads_count = ENV.fetch("RAILS_MAX_THREADS") { 5 } 8 | min_threads_count = ENV.fetch("RAILS_MIN_THREADS") { max_threads_count } 9 | threads min_threads_count, max_threads_count 10 | 11 | # Specifies the `port` that Puma will listen on to receive requests; default is 3000. 12 | # 13 | port ENV.fetch("PORT") { 3000 } 14 | 15 | # Specifies the `environment` that Puma will run in. 16 | # 17 | environment ENV.fetch("RAILS_ENV") { "development" } 18 | 19 | # Specifies the `pidfile` that Puma will use. 20 | pidfile ENV.fetch("PIDFILE") { "tmp/pids/server.pid" } 21 | 22 | # Specifies the number of `workers` to boot in clustered mode. 23 | # Workers are forked web server processes. If using threads and workers together 24 | # the concurrency of the application would be max `threads` * `workers`. 25 | # Workers do not work on JRuby or Windows (both of which do not support 26 | # processes). 27 | # 28 | # workers ENV.fetch("WEB_CONCURRENCY") { 2 } 29 | 30 | # Use the `preload_app!` method when specifying a `workers` number. 31 | # This directive tells Puma to first boot the application and load code 32 | # before forking the application. This takes advantage of Copy On Write 33 | # process behavior so workers use less memory. 34 | # 35 | # preload_app! 36 | 37 | # Allow puma to be restarted by `rails restart` command. 38 | plugin :tmp_restart 39 | -------------------------------------------------------------------------------- /public/500.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | We're sorry, but something went wrong (500) 5 | 6 | 55 | 56 | 57 | 58 | 59 |
60 |
61 |

We're sorry, but something went wrong.

62 |
63 |

If you are the application owner check the logs for more information.

64 |
65 | 66 | 67 | -------------------------------------------------------------------------------- /public/422.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | The change you wanted was rejected (422) 5 | 6 | 55 | 56 | 57 | 58 | 59 |
60 |
61 |

The change you wanted was rejected.

62 |

Maybe you tried to change something you didn't have access to.

63 |
64 |

If you are the application owner check the logs for more information.

65 |
66 | 67 | 68 | -------------------------------------------------------------------------------- /public/404.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | The page you were looking for doesn't exist (404) 5 | 6 | 55 | 56 | 57 | 58 | 59 |
60 |
61 |

The page you were looking for doesn't exist.

62 |

You may have mistyped the address or the page may have moved.

63 |
64 |

If you are the application owner check the logs for more information.

65 |
66 | 67 | 68 | -------------------------------------------------------------------------------- /config/environments/test.rb: -------------------------------------------------------------------------------- 1 | # The test environment is used exclusively to run your application's 2 | # test suite. You never need to work with it otherwise. Remember that 3 | # your test database is "scratch space" for the test suite and is wiped 4 | # and recreated between test runs. Don't rely on the data there! 5 | 6 | Rails.application.configure do 7 | # Settings specified here will take precedence over those in config/application.rb. 8 | 9 | config.cache_classes = false 10 | 11 | # Do not eager load code on boot. This avoids loading your whole application 12 | # just for the purpose of running a single test. If you are using a tool that 13 | # preloads Rails for running tests, you may have to set it to true. 14 | config.eager_load = false 15 | 16 | # Configure public file server for tests with Cache-Control for performance. 17 | config.public_file_server.enabled = true 18 | config.public_file_server.headers = { 19 | 'Cache-Control' => "public, max-age=#{1.hour.to_i}" 20 | } 21 | 22 | # Show full error reports and disable caching. 23 | config.consider_all_requests_local = true 24 | config.action_controller.perform_caching = false 25 | config.cache_store = :null_store 26 | 27 | # Raise exceptions instead of rendering exception templates. 28 | config.action_dispatch.show_exceptions = false 29 | 30 | # Disable request forgery protection in test environment. 31 | config.action_controller.allow_forgery_protection = false 32 | 33 | # Store uploaded files on the local file system in a temporary directory. 34 | config.active_storage.service = :test 35 | 36 | config.action_mailer.perform_caching = false 37 | 38 | # Tell Action Mailer not to deliver emails to the real world. 39 | # The :test delivery method accumulates sent emails in the 40 | # ActionMailer::Base.deliveries array. 41 | config.action_mailer.delivery_method = :test 42 | 43 | # Print deprecation notices to the stderr. 44 | config.active_support.deprecation = :stderr 45 | 46 | # Raises error for missing translations. 47 | # config.action_view.raise_on_missing_translations = true 48 | end 49 | -------------------------------------------------------------------------------- /app/javascript/packs/application.js: -------------------------------------------------------------------------------- 1 | // This file is automatically compiled by Webpack, along with any other files 2 | // present in this directory. You're encouraged to place your actual application logic in 3 | // a relevant structure within app/javascript and only use these pack files to reference 4 | // that code so it'll be compiled. 5 | 6 | require("@rails/ujs").start() 7 | require("turbolinks").start() 8 | require("@rails/activestorage").start() 9 | require("channels") 10 | 11 | 12 | // Uncomment to copy all static images under ../images to the output folder and reference 13 | // them with the image_pack_tag helper in views (e.g <%= image_pack_tag 'rails.png' %>) 14 | // or the `imagePath` JavaScript helper below. 15 | // 16 | // const images = require.context('../images', true) 17 | // const imagePath = (name) => images(name, true) 18 | 19 | 20 | document.addEventListener("turbolinks:load", function() { 21 | let timeouts = {} 22 | 23 | var token = document.getElementsByName('csrf-token')[0].content 24 | 25 | async function submitUpdatedWritablePage(txt, page_id, disclaimer){ 26 | console.log("txt", txt, "page_id", page_id); 27 | const response = await fetch(`/writable_pages/${page_id}.json`, { 28 | method: 'PUT', // or 'PUT' 29 | body: JSON.stringify({notes: txt, disclaimer:disclaimer }), 30 | headers: { 31 | 'Content-Type': 'application/json', 32 | 'X-CSRF-Token': token 33 | } 34 | }); 35 | const myJson = await response.json(); 36 | 37 | } 38 | 39 | document.addEventListener('keyup', function (event) { 40 | // If the clicked element doesn't have the right selector, bail 41 | if (!event.target.matches('.onkeyupdelay')) return; 42 | console.log(event.target.dataset.pageId) 43 | if (timeouts[event.target.dataset.pageId]){ 44 | clearTimeout(timeouts[event.target.dataset.pageId]) 45 | } 46 | timeouts[event.target.dataset.pageId] = setTimeout(() => submitUpdatedWritablePage(event.target.value, event.target.dataset.pageId, event.target.dataset.disclaimer), 3000); 47 | 48 | }, false); 49 | }); -------------------------------------------------------------------------------- /app/models/payer.rb: -------------------------------------------------------------------------------- 1 | class Payer < ApplicationRecord 2 | has_many :ads, primary_key: :name, foreign_key: :funding_entity 3 | has_many :ad_archive_report_pages, primary_key: :name, foreign_key: :disclaimer 4 | 5 | def min_spend 6 | ads.joins(:impressions).group(:archive_id).sum(:min_spend).values.reduce(&:+) 7 | end 8 | def min_spend 9 | ads.joins(:impressions).group(:archive_id).sum(:max_spend).values.reduce(&:+) 10 | end 11 | 12 | def advertisers 13 | # Page.where(page_id: ads.unscope(:order).select("distinct page_id")) # this is very slow. 14 | AdArchiveReport.where(kind: "lifelong").last.ad_archive_report_pages.where(disclaimer: 'BIDEN FOR PRESIDENT').map(&:page) 15 | end 16 | 17 | # def advertiser_spend 18 | # advertisers.map(&:spend).reduce(&:+) 19 | # end 20 | 21 | def min_impressions 22 | #ads.joins(:impressions).group(:ad_archive_id).max(:crawl_date).sum(:min_impressions) 23 | ads.joins(:impressions).sum(:min_impressions) 24 | end 25 | 26 | # TODO should go in a mixin. 27 | # TODO: should happen in SQL. 28 | def topic_breakdown 29 | breakdown = Hash[*ads.unscope(:order).joins(writable_ad: [{:ad_text => [{ad_topics: :topic}]}]).select("topic, sum(coalesce(ad_topics.proportion, cast(1.0 as double precision))) as proportion").group(:topic).map{|a| [a.topic, a.proportion]}.flatten] 30 | total = breakdown.values.reduce(&:+) 31 | breakdown_proportions = {} 32 | breakdown.each do | topic, amt | 33 | breakdown_proportions[topic] = amt.to_f / total 34 | end 35 | breakdown_proportions 36 | end 37 | 38 | def targeting_methods 39 | individual_methods = FbpacAd.connection.execute("select target, segment, count(*) as count from (select jsonb_array_elements(targets)->>'segment' as segment, jsonb_array_elements(targets)->>'target' as target from fbpac_ads WHERE #{Ad.send(:sanitize_sql_for_conditions, ["fbpac_ads.paid_for_by = ?", [name]] )}) q group by segment, target order by count desc").to_a 40 | combined_methods = FbpacAd.unscope(:order).where(paid_for_by: name).group(:targets).count.to_a.sort_by{|a, b| -b} 41 | {individual_methods: individual_methods, combined_methods: combined_methods} 42 | end 43 | 44 | 45 | end 46 | -------------------------------------------------------------------------------- /babel.config.js: -------------------------------------------------------------------------------- 1 | module.exports = function(api) { 2 | var validEnv = ['development', 'test', 'production'] 3 | var currentEnv = api.env() 4 | var isDevelopmentEnv = api.env('development') 5 | var isProductionEnv = api.env('production') 6 | var isTestEnv = api.env('test') 7 | 8 | if (!validEnv.includes(currentEnv)) { 9 | throw new Error( 10 | 'Please specify a valid `NODE_ENV` or ' + 11 | '`BABEL_ENV` environment variables. Valid values are "development", ' + 12 | '"test", and "production". Instead, received: ' + 13 | JSON.stringify(currentEnv) + 14 | '.' 15 | ) 16 | } 17 | 18 | return { 19 | presets: [ 20 | isTestEnv && [ 21 | require('@babel/preset-env').default, 22 | { 23 | targets: { 24 | node: 'current' 25 | } 26 | } 27 | ], 28 | (isProductionEnv || isDevelopmentEnv) && [ 29 | require('@babel/preset-env').default, 30 | { 31 | forceAllTransforms: true, 32 | useBuiltIns: 'entry', 33 | corejs: 3, 34 | modules: false, 35 | exclude: ['transform-typeof-symbol'] 36 | } 37 | ] 38 | ].filter(Boolean), 39 | plugins: [ 40 | require('babel-plugin-macros'), 41 | require('@babel/plugin-syntax-dynamic-import').default, 42 | isTestEnv && require('babel-plugin-dynamic-import-node'), 43 | require('@babel/plugin-transform-destructuring').default, 44 | [ 45 | require('@babel/plugin-proposal-class-properties').default, 46 | { 47 | loose: true 48 | } 49 | ], 50 | [ 51 | require('@babel/plugin-proposal-object-rest-spread').default, 52 | { 53 | useBuiltIns: true 54 | } 55 | ], 56 | [ 57 | require('@babel/plugin-transform-runtime').default, 58 | { 59 | helpers: false, 60 | regenerator: true, 61 | corejs: false 62 | } 63 | ], 64 | [ 65 | require('@babel/plugin-transform-regenerator').default, 66 | { 67 | async: false 68 | } 69 | ] 70 | ].filter(Boolean) 71 | } 72 | } 73 | -------------------------------------------------------------------------------- /app/models/page.rb: -------------------------------------------------------------------------------- 1 | class Page < ApplicationRecord 2 | self.primary_key = :page_id 3 | has_many :ads, primary_key: :page_id 4 | has_many :fbpac_ads, primary_key: :page_name, foreign_key: :advertiser 5 | has_one :writable_page, primary_key: :page_id, foreign_key: :page_id # just a proxy 6 | has_many :ad_archive_report_pages, primary_key: :page_id, foreign_key: :page_id 7 | has_many :writable_ads, primary_key: :page_id, foreign_key: :page_id 8 | def min_spend 9 | ads.joins(:impressions).group(:archive_id).sum(:min_spend).values.reduce(&:+) 10 | end 11 | def max_spend 12 | ads.joins(:impressions).group(:archive_id).sum(:max_spend).values.reduce(&:+) 13 | end 14 | 15 | def payers 16 | Payer.where(name: ads.unscope(:order).group("funding_entity").count.keys.reject{|a| a.blank?}) 17 | end 18 | 19 | def min_impressions 20 | ads.joins(:impressions).sum(:min_impressions) 21 | end 22 | 23 | def topic_breakdown 24 | breakdown = Hash[*writable_ads.joins({:ad_text => [{ad_topics: :topic}]}).select("topic, sum(coalesce(ad_topics.proportion, cast(1.0 as double precision))) as proportion").group(:topic).map{|a| [a.topic, a.proportion]}.flatten] 25 | 26 | total = breakdown.values.reduce(&:+) 27 | breakdown_proportions = {} 28 | breakdown.each do | topic, amt | 29 | breakdown_proportions[topic] = amt.to_f / total 30 | end 31 | breakdown_proportions 32 | end 33 | 34 | def notes=(text) 35 | if writable_page.nil? 36 | self.writable_page = WritablePage.new 37 | end 38 | writable_page.notes = text 39 | writable_page.save 40 | end 41 | 42 | def notes 43 | writable_page&.notes 44 | end 45 | 46 | def targeting_methods 47 | individual_methods = FbpacAd.connection.execute("select target, segment, count(*) as count from (select jsonb_array_elements(targets)->>'segment' as segment, jsonb_array_elements(targets)->>'target' as target from fbpac_ads WHERE #{Ad.send(:sanitize_sql_for_conditions, ["fbpac_ads.advertiser = ?", [page_name]] )}) q group by segment, target order by count desc").to_a 48 | combined_methods = FbpacAd.unscope(:order).where(advertiser: page_name).group(:targets).count.to_a.sort_by{|a, b| -b} 49 | {individual_methods: individual_methods, combined_methods: combined_methods} 50 | end 51 | 52 | 53 | 54 | def to_s 55 | "#" 56 | end 57 | 58 | end -------------------------------------------------------------------------------- /config/routes.rb: -------------------------------------------------------------------------------- 1 | Rails.application.routes.draw do 2 | devise_for :users 3 | devise_scope :user do 4 | get "users", to: "users/registrations#index" 5 | get "other_users/new", to: "users/registrations#sign_up_other_user" 6 | post "other_users/create", to: "users/registrations#create_other_user" 7 | delete "other_users/:user_id", to: "users/registrations#delete_other_user", as: "delete_other_user" 8 | end 9 | 10 | 11 | 12 | 13 | root to: "ads#overview" 14 | 15 | get "payers", to: "payers#index" 16 | get "payers/:id", to: "payers#show" 17 | get "payers_by_name/:payer_name", to: "payers#show", :payer_name => /[^\/]+(?=\.html\z|\.json\z)|[^\/]+/ 18 | 19 | 20 | get "pages", to: "pages#index" 21 | get "pages/:id", to: "pages#show" 22 | get "pages_by_name/:page_name", to: "pages#show", :page_name => /[^\/]+(?=\.html\z|\.json\z)|[^\/]+/ 23 | 24 | get 'writable_pages/update' 25 | put "writable_pages/:page_id", to: 'writable_pages#update' 26 | get "topics", to: "ads#topics" 27 | 28 | get "ads", to: "ads#index" 29 | get "ads/swing_state_ads", to: "ads#swing_state_ads" 30 | get "ads/search/", to: "ads#jeremys_double_method_search" 31 | get "ads/list_targets", to: "ads#list_targets" 32 | get "ads/pivot/:kind", to: "fbpac_ads#pivot" 33 | 34 | get "ads/:archive_id", to: "ads#show" 35 | get "ads_by_ad_id/:ad_id", to: "ads#show" 36 | get "ads_by_archive_id/:archive_id", to: "ads#show" 37 | get "ads_by_text/:text_hash", to: "ads#show_by_text" 38 | 39 | get "fbpac_ads/suppress", to: "fbpac_ads#suppress_page" 40 | post "fbpac_ads/suppress", to: "fbpac_ads#suppress" 41 | 42 | get "bigspenders", to: "pages#bigspenders" 43 | 44 | 45 | get "interim/youtube/", to: "youtube#index" 46 | get "interim/youtube/advertiser/:targ", to: "youtube#advertiser" 47 | get 'interim/youtube/targeting/:targ', to: "youtube#targeting" 48 | get 'interim/youtube/targeting_all/:targ', to: "youtube#targeting_all" 49 | get 'interim/youtube/advertiser_all/:targ', to: "youtube#advertiser_all" 50 | 51 | scope "pp" do 52 | get "ads", to: "fbpac_ads#index" 53 | get "ads/homepage_stats", to: "fbpac_ads#homepage_stats" 54 | get "ads/write_homepage_stats", to: "fbpac_ads#write_homepage_stats" 55 | get "ads/:id", to: "fbpac_ads#show" # okay 56 | get "persona", to: "fbpac_ads#persona" 57 | end 58 | 59 | 60 | end 61 | -------------------------------------------------------------------------------- /config/webpacker.yml: -------------------------------------------------------------------------------- 1 | # Note: You must restart bin/webpack-dev-server for changes to take effect 2 | 3 | default: &default 4 | source_path: app/javascript 5 | source_entry_path: packs 6 | public_root_path: public 7 | public_output_path: packs 8 | cache_path: tmp/cache/webpacker 9 | check_yarn_integrity: false 10 | webpack_compile_output: false 11 | 12 | # Additional paths webpack should lookup modules 13 | # ['app/assets', 'engine/foo/app/assets'] 14 | resolved_paths: [] 15 | 16 | # Reload manifest.json on all requests so we reload latest compiled packs 17 | cache_manifest: false 18 | 19 | # Extract and emit a css file 20 | extract_css: false 21 | 22 | static_assets_extensions: 23 | - .jpg 24 | - .jpeg 25 | - .png 26 | - .gif 27 | - .tiff 28 | - .ico 29 | - .svg 30 | - .eot 31 | - .otf 32 | - .ttf 33 | - .woff 34 | - .woff2 35 | 36 | extensions: 37 | - .mjs 38 | - .js 39 | - .sass 40 | - .scss 41 | - .css 42 | - .module.sass 43 | - .module.scss 44 | - .module.css 45 | - .png 46 | - .svg 47 | - .gif 48 | - .jpeg 49 | - .jpg 50 | 51 | development: 52 | <<: *default 53 | compile: false 54 | 55 | # Verifies that correct packages and versions are installed by inspecting package.json, yarn.lock, and node_modules 56 | check_yarn_integrity: true 57 | 58 | # Reference: https://webpack.js.org/configuration/dev-server/ 59 | dev_server: 60 | https: false 61 | host: localhost 62 | port: 3035 63 | public: localhost:3035 64 | hmr: false 65 | # Inline should be set to true if using HMR 66 | inline: true 67 | overlay: true 68 | compress: true 69 | disable_host_check: true 70 | use_local_ip: false 71 | quiet: false 72 | headers: 73 | 'Access-Control-Allow-Origin': '*' 74 | watch_options: 75 | ignored: '**/node_modules/**' 76 | 77 | 78 | test: 79 | <<: *default 80 | compile: true 81 | 82 | # Compile test packs to a separate directory 83 | public_output_path: packs-test 84 | 85 | production: 86 | <<: *default 87 | 88 | # Production depends on precompilation of packs prior to booting for performance. 89 | compile: false 90 | 91 | # Extract and emit a css file 92 | extract_css: true 93 | 94 | # Cache manifest.json for performance 95 | cache_manifest: true 96 | -------------------------------------------------------------------------------- /config/environments/development.rb: -------------------------------------------------------------------------------- 1 | Rails.application.configure do 2 | # Settings specified here will take precedence over those in config/application.rb. 3 | 4 | # In the development environment your application's code is reloaded on 5 | # every request. This slows down response time but is perfect for development 6 | # since you don't have to restart the web server when you make code changes. 7 | config.cache_classes = false 8 | 9 | # Do not eager load code on boot. 10 | config.eager_load = false 11 | 12 | # Show full error reports. 13 | config.consider_all_requests_local = true 14 | 15 | # Enable/disable caching. By default caching is disabled. 16 | # Run rails dev:cache to toggle caching. 17 | if Rails.root.join('tmp', 'caching-dev.txt').exist? 18 | config.action_controller.perform_caching = true 19 | config.action_controller.enable_fragment_cache_logging = true 20 | 21 | config.cache_store = :memory_store 22 | config.public_file_server.headers = { 23 | 'Cache-Control' => "public, max-age=#{2.days.to_i}" 24 | } 25 | else 26 | config.action_controller.perform_caching = false 27 | 28 | config.cache_store = :null_store 29 | end 30 | 31 | # Store uploaded files on the local file system (see config/storage.yml for options). 32 | config.active_storage.service = :local 33 | 34 | # Don't care if the mailer can't send. 35 | config.action_mailer.raise_delivery_errors = false 36 | 37 | config.action_mailer.perform_caching = false 38 | 39 | # Print deprecation notices to the Rails logger. 40 | config.active_support.deprecation = :log 41 | 42 | # Raise an error on page load if there are pending migrations. 43 | config.active_record.migration_error = :page_load 44 | 45 | # Highlight code that triggered database queries in logs. 46 | config.active_record.verbose_query_logs = true 47 | 48 | # Debug mode disables concatenation and preprocessing of assets. 49 | # This option may cause significant delays in view rendering with a large 50 | # number of complex assets. 51 | config.assets.debug = true 52 | 53 | # Suppress logger output for asset requests. 54 | config.assets.quiet = true 55 | 56 | config.action_mailer.default_url_options = { host: 'localhost', port: 3000 } 57 | 58 | # Raises error for missing translations. 59 | # config.action_view.raise_on_missing_translations = true 60 | 61 | # Use an evented file watcher to asynchronously detect changes in source code, 62 | # routes, locales, etc. This feature depends on the listen gem. 63 | config.file_watcher = ActiveSupport::EventedFileUpdateChecker 64 | end 65 | -------------------------------------------------------------------------------- /app/views/youtube/list.html.erb: -------------------------------------------------------------------------------- 1 | 2 | 3 | Interim YouTube Ads Dashboard 4 | 5 | 6 | Home: FB dashboard 7 |

Jeremy's Interim YouTube Ads Dashboard

8 | 9 |

<%= @matching_ads.size %> ads <%= @query %>

10 | 38 | 39 | 40 | 41 | -------------------------------------------------------------------------------- /app/controllers/payers_controller.rb: -------------------------------------------------------------------------------- 1 | class PayersController < ApplicationController 2 | 3 | def show 4 | @payer = params[:id] ? Payer.find(params[:id]) : Payer.find_by(name: params[:payer_name]) 5 | 6 | # count of ads 7 | @count_ads = @payer.ads.size 8 | 9 | # sum of min impressions for all ads 10 | @min_impressions = @payer.min_impressions 11 | 12 | # distinct advertisers (i.e. pages) 13 | @advertisers = @payer.advertisers 14 | 15 | # sum of spend for all advertisers 16 | # @min_spend = @payer.min_spend # removed because precise_spend is better 17 | 18 | starting_point_aarps = @payer.ad_archive_report_pages.where(ad_archive_report: AdArchiveReport.starting_point) 19 | 20 | aarps = @payer.ad_archive_report_pages.where(ad_archive_report: AdArchiveReport.where(loaded: true, kind: 'lifelong').order(:scrape_date).last) 21 | @precise_spend = aarps.sum(:amount_spent) - starting_point_aarps.sum(:amount_spent) 22 | @report_count_ads = aarps.sum(:ads_count) 23 | 24 | # breakdown of topics for all ads. 25 | @topics = @payer.topic_breakdown 26 | 27 | # TODO: count of ads with a CollectorAd 28 | fbpac_ads = FbpacAd.where(paid_for_by: @payer.name) 29 | @fbpac_ads_cnt = fbpac_ads.count 30 | # TODO: targetings used 31 | @targetings = @payer.targeting_methods 32 | 33 | # TODO: targetings used 34 | 35 | # TODO: domain names linked to in ads (TODO: has to come from FBPAC or AdLibrary collector) 36 | 37 | # TODO: notes are appended (or longest of?) unique payer / page pairs. 38 | 39 | respond_to do |format| 40 | format.html 41 | format.json { render json: { 42 | payer: @payer.name, 43 | notes: @payer.notes, 44 | 45 | ads: @count_ads, 46 | fbpac_ads: @fbpac_ads_cnt, 47 | advertisers: @advertisers, 48 | 49 | min_impressions: @min_impressions, 50 | # min_spend: @min_spend, 51 | precise_spend: @precise_spend, 52 | topics: @topics, 53 | targetings: @targetings 54 | 55 | } } 56 | end 57 | 58 | end 59 | 60 | def index 61 | # lists all known payers 62 | @all = Payer.all 63 | respond_to do |format| 64 | format.html 65 | format.json { render json: @all } 66 | end 67 | end 68 | 69 | def index 70 | # lists all known payers 71 | @payers = Payer.paginate(page: params[:page], per_page: 30) 72 | 73 | 74 | respond_to do |format| 75 | format.html 76 | format.json { render json: { 77 | payers: @payers, 78 | total_ads: @payers.total_entries, 79 | n_pages: @payers.total_pages, 80 | page: params[:page] || 1 81 | } 82 | } 83 | end 84 | end 85 | 86 | 87 | end 88 | -------------------------------------------------------------------------------- /Gemfile: -------------------------------------------------------------------------------- 1 | source 'https://rubygems.org' 2 | git_source(:github) { |repo| "https://github.com/#{repo}.git" } 3 | 4 | ruby '2.6.5' 5 | gem 'actionpack-action_caching' 6 | gem 'active_record_extended' 7 | gem 'aws-sdk-s3', '~> 1' 8 | 9 | gem 'will_paginate', '~> 3.1.0' 10 | gem 'pg' 11 | gem "pg_search" 12 | 13 | gem 'elasticsearch-model', github: 'elastic/elasticsearch-rails', branch: '6.x' 14 | gem 'elasticsearch-rails', github: 'elastic/elasticsearch-rails', branch: '6.x' 15 | gem 'elasticsearch-dsl', github: 'elastic/elasticsearch-ruby', branch: '6.x' 16 | 17 | gem "couchrest" 18 | 19 | gem "rest-client" 20 | gem "ruby-progressbar" 21 | gem "selenium" 22 | gem "webdrivers" 23 | gem 'devise' 24 | gem 'rack-cors' 25 | 26 | # Bundle edge Rails instead: gem 'rails', github: 'rails/rails' 27 | gem 'rails', '~> 6.0.0' 28 | # Use sqlite3 as the database for Active Record 29 | gem 'sqlite3', '~> 1.4' 30 | # Use Puma as the app server 31 | gem 'puma', '~> 3.11' 32 | # Use SCSS for stylesheets 33 | #gem 'sass-rails', '~> 5' 34 | gem 'sassc-rails' 35 | # Transpile app-like JavaScript. Read more: https://github.com/rails/webpacker 36 | gem 'webpacker', '~> 4.0' 37 | # Turbolinks makes navigating your web application faster. Read more: https://github.com/turbolinks/turbolinks 38 | gem 'turbolinks', '~> 5' 39 | # Build JSON APIs with ease. Read more: https://github.com/rails/jbuilder 40 | gem 'jbuilder', '~> 2.7' 41 | # Use Redis adapter to run Action Cable in production 42 | # gem 'redis', '~> 4.0' 43 | # Use Active Model has_secure_password 44 | # gem 'bcrypt', '~> 3.1.7' 45 | 46 | # Use Active Storage variant 47 | # gem 'image_processing', '~> 1.2' 48 | 49 | # Reduces boot times through caching; required in config/boot.rb 50 | gem 'bootsnap', '>= 1.4.2', require: false 51 | 52 | group :development, :test do 53 | # Call 'byebug' anywhere in the code to stop execution and get a debugger console 54 | gem 'byebug', platforms: [:mri, :mingw, :x64_mingw] 55 | end 56 | 57 | group :development do 58 | # Access an interactive console on exception pages or by calling 'console' anywhere in the code. 59 | gem 'web-console', '>= 3.3.0' 60 | gem 'listen', '>= 3.0.5', '< 3.2' 61 | # Spring speeds up development by keeping your application running in the background. Read more: https://github.com/rails/spring 62 | gem 'spring' 63 | gem 'spring-watcher-listen', '~> 2.0.0' 64 | end 65 | 66 | group :test do 67 | # Adds support for Capybara system testing and selenium driver 68 | gem 'capybara', '>= 2.15' 69 | gem 'selenium-webdriver' 70 | # Easy installation and use of web drivers to run system tests with browsers 71 | gem 'webdrivers' 72 | end 73 | 74 | # Windows does not include zoneinfo files, so bundle the tzinfo-data gem 75 | gem 'tzinfo-data', platforms: [:mingw, :mswin, :x64_mingw, :jruby] 76 | -------------------------------------------------------------------------------- /app/controllers/pages_controller.rb: -------------------------------------------------------------------------------- 1 | class PagesController < ApplicationController 2 | 3 | 4 | def show 5 | # 153080620724 is Trump 6 | # FBPAC ads only sometimes have a page_id 7 | 8 | @page = params[:id] ? Page.find_by(page_id: params[:id]) : Page.find_by(page_name: params[:page_name]) 9 | 10 | # count of ads 11 | @count_ads = AdArchiveReportPage.where(ad_archive_report_id: AdArchiveReport.where(kind: 'lifelong', loaded: true).order(:scrape_date).last.id, page_id: @page.page_id).sum("ads_count") 12 | 13 | 14 | # sum of min impressions for all ads 15 | # @min_impressions = @page.min_impressions 16 | @min_impressions = "TK" 17 | 18 | # count of distinct payers 19 | @payers = @page.payers 20 | 21 | # sum of spend for all payers 22 | # @min_spend = @page.min_spend # removed because precise_spend is better 23 | # @max_spend = @page.max_spend # removed because precise_spend is better 24 | aarps = @page.ad_archive_report_pages.where(ad_archive_report: AdArchiveReport.where(loaded: true, kind: 'lifelong').order(:scrape_date).last) 25 | starting_point_aarps = @page.ad_archive_report_pages.where(ad_archive_report: AdArchiveReport.starting_point) 26 | 27 | @precise_spend = aarps.sum(:amount_spent) - starting_point_aarps.sum(:amount_spent) 28 | @report_count_ads = aarps.sum(:ads_count) 29 | 30 | # breakdown of topics for all ads. 31 | @topics = @page.topic_breakdown 32 | 33 | # count of ads with a CollectorAd 34 | fbpac_ads = FbpacAd.where(advertiser: @page.page_name) 35 | @fbpac_ads_cnt = fbpac_ads.count 36 | # TODO: targetings used 37 | @targetings = @page.targeting_methods 38 | 39 | # TODO: domain names linked to in ads (TODO: has to come from FBPAC or AdLibrary collector) 40 | 41 | respond_to do |format| 42 | format.html 43 | format.json { render json: { 44 | page: @page.page_name, 45 | notes: @page.writable_page&.notes, 46 | 47 | ads: @count_ads, 48 | fbpac_ads: @fbpac_ads_cnt, 49 | payers: @payers, 50 | 51 | min_impressions: @min_impressions, 52 | # min_spend: @min_spend, 53 | # max_spend: @max_spend, 54 | precise_spend: @precise_spend, 55 | topics: @topics, 56 | 57 | targetings: @targetings 58 | } } 59 | end 60 | end 61 | 62 | def bigspenders 63 | @big_spenders = BigSpender.preload(:writable_page).preload(:ad_archive_report_page).preload(:page).order("spend_amount desc") 64 | respond_to do |format| 65 | format.html 66 | format.json { render json: @big_spenders } 67 | end 68 | end 69 | 70 | def index 71 | # lists all known payers 72 | @pages = Page.paginate(page: params[:page], per_page: 30) 73 | 74 | 75 | respond_to do |format| 76 | format.html 77 | format.json { render json: { 78 | pages: @pages, 79 | total_ads: @pages.total_entries, 80 | n_pages: @pages.total_pages, 81 | page: params[:page] || 1 82 | } 83 | } 84 | 85 | end 86 | end 87 | 88 | end 89 | -------------------------------------------------------------------------------- /bin/bundle: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env ruby 2 | # frozen_string_literal: true 3 | 4 | # 5 | # This file was generated by Bundler. 6 | # 7 | # The application 'bundle' is installed as part of a gem, and 8 | # this file is here to facilitate running it. 9 | # 10 | 11 | require "rubygems" 12 | 13 | m = Module.new do 14 | module_function 15 | 16 | def invoked_as_script? 17 | File.expand_path($0) == File.expand_path(__FILE__) 18 | end 19 | 20 | def env_var_version 21 | ENV["BUNDLER_VERSION"] 22 | end 23 | 24 | def cli_arg_version 25 | return unless invoked_as_script? # don't want to hijack other binstubs 26 | return unless "update".start_with?(ARGV.first || " ") # must be running `bundle update` 27 | bundler_version = nil 28 | update_index = nil 29 | ARGV.each_with_index do |a, i| 30 | if update_index && update_index.succ == i && a =~ Gem::Version::ANCHORED_VERSION_PATTERN 31 | bundler_version = a 32 | end 33 | next unless a =~ /\A--bundler(?:[= ](#{Gem::Version::VERSION_PATTERN}))?\z/ 34 | bundler_version = $1 || ">= 0.a" 35 | update_index = i 36 | end 37 | bundler_version 38 | end 39 | 40 | def gemfile 41 | gemfile = ENV["BUNDLE_GEMFILE"] 42 | return gemfile if gemfile && !gemfile.empty? 43 | 44 | File.expand_path("../../Gemfile", __FILE__) 45 | end 46 | 47 | def lockfile 48 | lockfile = 49 | case File.basename(gemfile) 50 | when "gems.rb" then gemfile.sub(/\.rb$/, gemfile) 51 | else "#{gemfile}.lock" 52 | end 53 | File.expand_path(lockfile) 54 | end 55 | 56 | def lockfile_version 57 | return unless File.file?(lockfile) 58 | lockfile_contents = File.read(lockfile) 59 | return unless lockfile_contents =~ /\n\nBUNDLED WITH\n\s{2,}(#{Gem::Version::VERSION_PATTERN})\n/ 60 | Regexp.last_match(1) 61 | end 62 | 63 | def bundler_version 64 | @bundler_version ||= begin 65 | env_var_version || cli_arg_version || 66 | lockfile_version || "#{Gem::Requirement.default}.a" 67 | end 68 | end 69 | 70 | def load_bundler! 71 | ENV["BUNDLE_GEMFILE"] ||= gemfile 72 | 73 | # must dup string for RG < 1.8 compatibility 74 | activate_bundler(bundler_version.dup) 75 | end 76 | 77 | def activate_bundler(bundler_version) 78 | if Gem::Version.correct?(bundler_version) && Gem::Version.new(bundler_version).release < Gem::Version.new("2.0") 79 | bundler_version = "< 2" 80 | end 81 | gem_error = activation_error_handling do 82 | gem "bundler", bundler_version 83 | end 84 | return if gem_error.nil? 85 | require_error = activation_error_handling do 86 | require "bundler/version" 87 | end 88 | return if require_error.nil? && Gem::Requirement.new(bundler_version).satisfied_by?(Gem::Version.new(Bundler::VERSION)) 89 | warn "Activating bundler (#{bundler_version}) failed:\n#{gem_error.message}\n\nTo install the version of bundler this project requires, run `gem install bundler -v '#{bundler_version}'`" 90 | exit 42 91 | end 92 | 93 | def activation_error_handling 94 | yield 95 | nil 96 | rescue StandardError, LoadError => e 97 | e 98 | end 99 | end 100 | 101 | m.load_bundler! 102 | 103 | if m.invoked_as_script? 104 | load Gem.bin_path("bundler", "bundle") 105 | end 106 | -------------------------------------------------------------------------------- /lib/tasks/swing_state_ads.rake: -------------------------------------------------------------------------------- 1 | 2 | 3 | SWING_STATES = ['Michigan', 'Wisconsin', 'Pennsylvania', 'Florida', 'Arizona', 'North Carolina', 'Nebraska'] 4 | LEANS = ["Maine", "Minnesota", "New Hampshire", "Georgia", "Texas"] 5 | # per Cook political report, 4/22/2020 https://cookpolitical.com/sites/default/files/2020-03/EC%20030920.4.pdf 6 | # Nebraska is just one CD, of course. 7 | 8 | namespace :swing_states do 9 | task get: :environment do 10 | start = Time.now 11 | count = 0 12 | 13 | old_swing_state_ads = Set.new(WritableAd.where(swing_state_ad: true).select("archive_id").map(&:archive_id)) 14 | 15 | advertiser_new_swing_ads_count = {} 16 | 17 | WritablePage.where(core: true).each do |wpage| 18 | resps = Ad.connection.execute("select ads.archive_id, ad_creative_body, sum(spend_percentage), array_to_json(array_agg(case when spend_percentage > 0.05 then region else null end order by region)) as states from ads join region_impressions using (archive_id) join impressions using (archive_id) where region in ('#{SWING_STATES.join("','")}') and spend_percentage < 0.90 and page_id = #{wpage.page_id} and min_impressions > 0 and ad_creation_time > '#{(Date.today - 60).to_s}' and (ad_delivery_stop_time is null or ad_delivery_stop_time > '#{(Date.today - 30).to_s}') group by ads.archive_id, ad_creative_body having sum(spend_percentage) > 0.8").to_a 19 | puts "found #{resps.size} for #{wpage.page.page_name}" 20 | resps.each do |ad_row| 21 | count += 1 22 | wad = WritableAd.find_by(archive_id: ad_row["archive_id"]) 23 | if wad.nil? 24 | ad = Ad.find_by(archive_id: ad_row["archive_id"]) 25 | wad = ad.create_writable_ad! 26 | ad.create_ad_text!(wad) 27 | end 28 | unless wad.swing_state_ad 29 | advertiser_new_swing_ads_count[wad.page_id] ||= 0 30 | advertiser_new_swing_ads_count[wad.page_id] += 1 31 | end 32 | old_swing_state_ads.delete(wad.archive_id) 33 | wad.swing_state_ad = true 34 | wad.states = JSON.load(ad_row["states"]).compact 35 | wad.save! 36 | wad.copy_screenshot_to_s3! 37 | end 38 | end 39 | 40 | stopped_being_swing_state_ads = old_swing_state_ads.count 41 | WritableAd.where(archive_id: stopped_being_swing_state_ads).update_all(swing_state_ad: false) 42 | 43 | new_advertisers_text = advertiser_new_swing_ads_count.map{|page_id, cnt| "#{Page.find(page_id).page_name}: #{cnt}"}.join(", ") 44 | 45 | msg = "(7/7): found #{count} swing state ads; #{stopped_being_swing_state_ads} stopped being swing state ads; #{new_advertisers_text}" 46 | puts msg 47 | 48 | job = Job.find_by(name: "swing_states:get") 49 | job_run = job.job_runs.create({ 50 | start_time: start, 51 | end_time: Time.now, 52 | success: true, 53 | }) 54 | 55 | RestClient.post( 56 | ENV["SLACKWH"], 57 | JSON.dump({"text" => msg }), 58 | {:content_type => "application/json"} 59 | ) if count > 0 && ENV["SLACKWH"] 60 | end 61 | task clear: :environment do 62 | # WritableAd.update_all(swing_state_ad: false) 63 | # WritableAd.update_all(states: []) 64 | # RestClient.post( 65 | # ENV["SLACKWH"], 66 | # JSON.dump({"text" => "(7/8): cleared swing state ads."}), 67 | # {:content_type => "application/json"} 68 | # ) if ENV["SLACKWH"] 69 | end 70 | end 71 | -------------------------------------------------------------------------------- /app/controllers/users/registrations_controller.rb: -------------------------------------------------------------------------------- 1 | # frozen_string_literal: true 2 | 3 | class Users::RegistrationsController < Devise::RegistrationsController 4 | # before_action :configure_sign_up_params, only: [:create] 5 | # before_action :configure_account_update_params, only: [:update] 6 | before_action :authenticate_user! 7 | 8 | # GET /resource/sign_up 9 | # def new 10 | # super 11 | # end 12 | 13 | # POST /resource 14 | # def create 15 | # super 16 | # end 17 | 18 | # GET /resource/edit 19 | def edit 20 | super 21 | end 22 | 23 | def index 24 | @users = User.all 25 | respond_with @users, template: "users/registrations/index" 26 | end 27 | 28 | def sign_up_other_user 29 | self.resource = User.new_with_session({}, session) 30 | # yield resource if block_given? 31 | respond_with resource, template: "users/registrations/new_other" 32 | end 33 | 34 | def create_other_user 35 | build_resource(sign_up_params) 36 | 37 | resource.save 38 | yield resource if block_given? 39 | if resource.persisted? 40 | if resource.active_for_authentication? 41 | set_flash_message! :notice, :signed_up 42 | sign_up(resource_name, resource) 43 | @users = User.all 44 | render template: "users/registrations/index" 45 | else 46 | set_flash_message! :notice, :"signed_up_but_#{resource.inactive_message}" 47 | expire_data_after_sign_in! 48 | respond_with resource, location: after_inactive_sign_up_path_for(resource) 49 | end 50 | else 51 | clean_up_passwords resource 52 | set_minimum_password_length 53 | set_flash_message! :notice, "failed" 54 | @users = User.all 55 | render template: "users/registrations/index" 56 | end 57 | end 58 | 59 | def delete_other_user 60 | resource = resource_class.find(params[:user_id]) 61 | resource.destroy 62 | Devise.sign_out_all_scopes ? sign_out : sign_out("user") 63 | set_flash_message! :notice, :destroyed 64 | yield user if block_given? 65 | @users = User.all 66 | render template: "users/registrations/index" 67 | end 68 | 69 | # PUT /resource 70 | # def update 71 | # super 72 | # end 73 | 74 | # DELETE /resource 75 | # def destroy 76 | # super 77 | # end 78 | 79 | # GET /resource/cancel 80 | # Forces the session data which is usually expired after sign 81 | # in to be expired now. This is useful if the user wants to 82 | # cancel oauth signing in/up in the middle of the process, 83 | # removing all OAuth session data. 84 | # def cancel 85 | # super 86 | # end 87 | 88 | # protected 89 | 90 | # If you have extra params to permit, append them to the sanitizer. 91 | # def configure_sign_up_params 92 | # devise_parameter_sanitizer.permit(:sign_up, keys: [:attribute]) 93 | # end 94 | 95 | # If you have extra params to permit, append them to the sanitizer. 96 | # def configure_account_update_params 97 | # devise_parameter_sanitizer.permit(:account_update, keys: [:attribute]) 98 | # end 99 | 100 | # The path used after sign up. 101 | def after_sign_up_path_for(resource) 102 | @users = User.all 103 | render template: "users/registrations/index" 104 | end 105 | 106 | # The path used after sign up for inactive accounts. 107 | # def after_inactive_sign_up_path_for(resource) 108 | # super(resource) 109 | # end 110 | end 111 | -------------------------------------------------------------------------------- /app/models/ad_text.rb: -------------------------------------------------------------------------------- 1 | class AdText < ApplicationRecord 2 | has_many :writable_ads, primary_key: :text_hash, foreign_key: :text_hash 3 | has_many :ad_topics 4 | has_many :topics, through: :ad_topics 5 | has_many :ads, through: :writable_ads 6 | has_many :fbpac_ads, through: :writable_ads 7 | has_many :impressions, through: :ads 8 | 9 | include PgSearch::Model 10 | 11 | pg_search_scope :search_for, 12 | against: %i(search_text), 13 | ignoring: :accents, 14 | ranked_by: "id", 15 | using: { 16 | tsearch: { 17 | negation: true, 18 | dictionary: "english", 19 | tsvector_column: 'tsv' 20 | } 21 | } 22 | 23 | def as_json(options) 24 | preset_options = { 25 | include: {writable_ads: {include: [:fbpac_ad]}, topics: {}} 26 | } 27 | if options[:include].is_a? Symbol 28 | options[:include] = Hash[options[:include], nil] 29 | end 30 | if !options[:include].nil? && options.has_key?(:include) 31 | options[:include] = preset_options[:include].deep_merge(!options.nil? && options[:include] ? options[:include] : {}) 32 | end 33 | 34 | # grab *one* the ad and fbpac ad 35 | # holds onto text hash, I guess. 36 | json = super(options) 37 | fbpac_ad = json["writable_ads"].find{|wad| wad.has_key?("fbpac_ad")}&.dig("fbpac_ad") || {} 38 | fbapi_ad_id = json["writable_ads"].find{|wad| wad["archive_id"]}&.dig("archive_id") 39 | if options[:ads] 40 | fbapi_ad = fbapi_ad_id ? options[:ads].find{|ad| ad.archive_id == fbapi_ad_id}.as_json(includes: []) : {} 41 | else 42 | fbapi_ad = fbapi_ad_id ? Ad.find(fbapi_ad_id).as_json(includes: []) : {} 43 | end 44 | 45 | topics = json.extract!("topics") 46 | new_json = json["writable_ads"].first.dup.without("ad", "fbpac_ad").merge(fbpac_ad.merge(fbapi_ad)).merge(topics) 47 | new_json["created_at"] = json["first_seen"] 48 | new_json["updated_at"] = json["last_seen"] 49 | new_json["page_id"] = json["page_id"] 50 | new_json["advertiser"] = json["advertiser"] 51 | new_json["paid_for_by"] = json["paid_for_by"] 52 | 53 | # Ad lookups (from the HL server takes 130ms each) 54 | # So we only do it if we dont' have any FBPAC examples. 55 | new_json["variants"] = json["writable_ads"].select{|wad| wad.has_key?("fbpac_ad")}.first(3).map{|ad| ad["fbpac_ad"]} 56 | new_json["variants"] = json["writable_ads"].select{|ad| ad["archive_id"]}.first(1).map do |ad| 57 | if options[:ads] 58 | options[:ads].find{|other_ad| other_ad.archive_id == ad["archive_id"]} 59 | else 60 | Ad.find(ad["archive_id"]) 61 | end 62 | end if new_json["variants"].size == 0 63 | 64 | new_json 65 | end 66 | 67 | # def self.jsonify(ad_text, fbpac_ads, api_ads) 68 | # json = ad_text.writable_ads.first 69 | # json["topics"] = ad_text.topics 70 | # json["variants"] = ad_text.writable_ads 71 | 72 | # end 73 | 74 | def self.classify_topic(ad_texts) 75 | res_json = RestClient.post(ENV["TOPICS_URL"] + "/topics", {'texts' => ad_texts.map(&:text)}.to_json, {content_type: :json, accept: :json}) 76 | res = JSON.parse(res_json) 77 | 78 | ad_texts.zip(res).each do |ad_text, topics| 79 | if topics.size == 0 80 | topics << "none" 81 | end 82 | ad_text.topics = topics.map{|t| Topic.find_or_create_by(topic: t)} 83 | end 84 | end 85 | end 86 | 87 | -------------------------------------------------------------------------------- /app/models/ad.rb: -------------------------------------------------------------------------------- 1 | 2 | 3 | class Ad < ApplicationRecord 4 | self.primary_key = 'archive_id' 5 | # has_one :collector_ad, primary_key: :archive_id # for the NEW collector, not existing FBPAC. 6 | belongs_to :payer, foreign_key: :name, primary_key: :funding_entity 7 | belongs_to :page, primary_key: :page_id 8 | has_one :impressions_record, primary_key: :archive_id, foreign_key: :archive_id 9 | default_scope { order(:archive_id) } 10 | 11 | has_one :writable_ad, primary_key: :archive_id, foreign_key: :archive_id # just a proxy 12 | has_one :fbpac_ad, primary_key: :ad_id, foreign_key: :id # doesn't work anymore, sadly 13 | 14 | def as_json(options={}) 15 | preset_options = { 16 | include: { page: { only: :page_name }, 17 | payer: { only: :name }, 18 | writable_ad: {topics: { only: :topic }} 19 | } 20 | } 21 | if options[:include].is_a? Symbol 22 | options[:include] = Hash[options[:include], nil] 23 | end 24 | if !options[:include].nil? && options.has_key?(:include) 25 | options[:include] = preset_options[:include].deep_merge(!options.nil? && options[:include] ? options[:include] : {}) 26 | end 27 | 28 | super(options).tap do |json| 29 | advertiser = (json["page"] || {})["page_name"] 30 | json["advertiser"] = advertiser if advertiser 31 | json["text"] = json.delete("ad_creative_body") 32 | json.delete("page") 33 | json["funding_entity"] = json["funding_entity"] || (json["payer"] || {})["name"] 34 | json["topics"] = json["topics"]&.map{|topic| topic["topic"]} 35 | json = json.delete("writable_ad") if json.has_key?("writable_ad") 36 | json = json.merge(json) 37 | end 38 | end 39 | 40 | def min_spend 41 | impressions.first.min_spend 42 | end 43 | 44 | def min_impressions 45 | impressions.first.min_impressions 46 | end 47 | 48 | def domain 49 | # has to come from collector ads or AdLibrary ads. 50 | end 51 | 52 | # # Exclude snapshot_url, is_active from JSON responses 53 | # def serializable_hash(options={}) 54 | # options = { 55 | # exclude: [:snapshot_url, :is_active] 56 | # }.update(options) 57 | # super(options) 58 | # end 59 | def text 60 | [ad_creative_body, ad_creative_link_caption, ad_creative_link_title, ad_creative_link_description].join(' ') 61 | end 62 | 63 | def clean_text 64 | # clean_text exists to get hashed. those hashes have to match with hashes from FBPAC-collected ads. 65 | # FBPAC ads only make it easy/possible to get the equivalent of ad_creative_body 66 | ad_creative_body.to_s.strip.downcase.gsub(/\s+/, ' ').gsub(/[^a-z 0-9]/, '') 67 | end 68 | 69 | def create_writable_ad! 70 | wad = WritableAd.new 71 | wad.archive_id = archive_id 72 | wad.page_id = page_id 73 | wad.save! 74 | wad 75 | end 76 | 77 | def create_ad_text!(wad) 78 | wad.text_hash = Digest::SHA1.hexdigest(clean_text) 79 | ad_text = AdText.find_or_create_by(text_hash: wad.text_hash) 80 | ad_text.text ||= text 81 | ad_text.search_text ||= page.page_name + " " + text 82 | ad_text.first_seen = [ad_text.first_seen, ad_creation_time].compact.min # set the creation time to be the earliest we've seen for this text. 83 | ad_text.last_seen = [ad_text.last_seen, ad_delivery_stop_time].compact.max 84 | ad_text.page_id ||= page_id 85 | ad_text.advertiser ||= page.page_name 86 | ad_text.paid_for_by ||= funding_entity 87 | ad_text.save! 88 | ad_text 89 | end 90 | 91 | 92 | end 93 | 94 | 95 | # select * from impressions join ads on ads.archive_id = impressions.archive_id where ads.archive_id = 625389131321564; 96 | # Ad.joins(:ad_topics).joins(:topics).sum("coalesce(ad_topics.proportion, cast(1.0 as double precision))") -------------------------------------------------------------------------------- /app/models/fbpac_ad.rb: -------------------------------------------------------------------------------- 1 | class FbpacAd < ApplicationRecord 2 | # belongs_to :ad, primary_key: :ad_id, foreign_key: :id # doesn't work anymore :( 3 | 4 | belongs_to :writable_ad, primary_key: :ad_id, foreign_key: :id 5 | delegate :ad_text, :to => :writable_ad, :allow_nil => true 6 | alias_method :as_propublica_json, :as_json 7 | 8 | 9 | # THIS DOES NOT IN FACT GET CALLED 10 | def as_json(options={}) 11 | # translating this schema to match the FB one as much as possible 12 | super.tap do |json| 13 | json["ad_creation_time"] = json.delete("created_at") 14 | json["text"] = json.delete("message") # TODO: remove HTML tags 15 | json["funding_entity"] = json["paid_for_by"] 16 | # what if page_id doesn't exist?! 17 | # json["page_id"] 18 | json["start_date"] = json.delete("created_at") 19 | json = json.merge(json) 20 | end 21 | end 22 | 23 | 24 | 25 | # MISSING_STR = "missingpaidforby" 26 | 27 | # def as_indexed_json(options={}) # for ElasticSearch 28 | # json = self.as_json() # TODO: this needs a lot of work, I don't know the right way to do this, presumably I'll want writablefbpacads too 29 | # # json["topics"] = json["topics"]&.map{|topic| topic["topic"]} 30 | # json["paid_for_by"] = MISSING_STR if (json["paid_for_by"].nil? || json["paid_for_by"].empty?) && json["ad_creation_time"] && json["ad_creation_time"]> "2018-07-01" 31 | # json 32 | # end 33 | 34 | def text 35 | Nokogiri::HTML(message).text.strip 36 | end 37 | def clean_text 38 | text.downcase.gsub(/\s+/, ' ').gsub(/[^a-z 0-9]/, '') 39 | end 40 | 41 | TARGETS_MAPPING = { 42 | # List => “You’re on a list” 43 | # Activity on the Facebook Family => “Activity on Facebook” 44 | # Retargeting => “You're similar to another group” 45 | 46 | } 47 | def self.correct_targets(targets_arr) 48 | 49 | end 50 | 51 | 52 | USERS_COUNT = 2424 + 5551 53 | def self.calculate_homepage_stats(lang) # internal only! 54 | political_ads_count = FbpacAd.where(lang: lang).where("political_probability > 0.70").count 55 | political_ads_today = FbpacAd.where(lang: lang).where("political_probability > 0.70").unscope(:order).where("created_at AT TIME ZONE 'America/New_York' > now() - interval '1 day' ").count 56 | starting_count = 187378 # select count(*) from fbpac_ads where created_at AT TIME ZONE 'America/New_York' <= '2020-01-01' and political_probability > 0.7 and suppressed = false and lang = 'en-US'; 57 | cumulative_political_ads_per_week = FbpacAd.unscope(:order).where(lang: lang).where("political_probability > 0.70").where("created_at AT TIME ZONE 'America/New_York' > '2020-01-01'").group("date_trunc('week', created_at AT TIME ZONE 'America/New_York')").select("count(*) as total, date_trunc('week', created_at AT TIME ZONE 'America/New_York') as week").sort_by{|ad| ad.week}.reduce([]){|memo, ad| memo << [ad.week, (memo.last ? memo.last[1] : starting_count) + ad.total]; memo} 58 | 59 | { 60 | user_count: USERS_COUNT, 61 | political_ads_total: political_ads_count, 62 | political_ads_today: political_ads_today, 63 | political_ads_per_day: cumulative_political_ads_per_week 64 | } 65 | end 66 | 67 | def create_writable_ad! 68 | wad = WritableAd.new 69 | wad.fbpac_ad = self 70 | wad.text_hash = Digest::SHA1.hexdigest(clean_text) 71 | wad.ad_text = create_ad_text!(wad) 72 | wad.save! 73 | wad 74 | end 75 | 76 | def create_ad_text!(wad) 77 | ad_text = AdText.find_or_initialize_by(text_hash: wad.text_hash) 78 | ad_text.text ||= text 79 | ad_text.search_text ||= advertiser.to_s + " " + text # TODO: consider adding CTA text, etc. 80 | 81 | ad_text.first_seen = [ad_text.first_seen, created_at].compact.min # set the creation time to be the earliest we've seen for this text. 82 | ad_text.last_seen = [ad_text.last_seen, updated_at].compact.max 83 | ad_text.advertiser ||= advertiser 84 | ad_text.paid_for_by ||= paid_for_by 85 | 86 | ad_text.save! 87 | ad_text 88 | end 89 | 90 | 91 | end 92 | -------------------------------------------------------------------------------- /app/views/ads/swing_state_ads.html.erb: -------------------------------------------------------------------------------- 1 |

Quartz Political Facebook Ads Dashboard

2 | 3 |

THIS IS A MOCKUP

4 | 5 |

Go ahead, persuade me: how campaigns are trying to persuade swing voters

6 | 7 |

Joe Biden and Donald Trump are spending many millions of dollars on online ads in hopes of winning the 2020 presidential election. So are outside groups. But a lot of that spending isn't directly going to convince voters -- it's meant to raise money (or collect your email address, that the candidates will later use to ask your for money).

8 | 9 |

Messaging differs between persuasive ads and fundraising ads, the messaging can differ. Fundraising targets get fed red meat on guns, abortion, the latest outrage.

10 | 11 |

On the other hand, persuasion targets are voters who are on the fence about who to vote for. Turnout targets are not very likely to vote -- but if they do, they'll definitely vote for one side or the other. Both groups may not be super plugged into the news.

12 | 13 |

That's where you'll see the first hints of new messages: Trump trying to paint Biden as too close to China, for instance. LINKTK.

14 | 15 |

We've highlighted ads we're pretty sure are meant for either persuasion or turnout targets. Ordinarily, you'd only see this if you live in a swing state and turn on the TV -- or if you are someone whose personal data indicates you're a persuasion or turnout target.

16 | 17 |

Quartz has analyzed Facebook's massive database of political ads and filtered out just the ads from major political spenders that are targeted just to swing states. These are probably the ads meant to persuade someone to turn out or to vote for one particular side. Fundraising targets can be anywhere in the country; a Rhode Island dollar is the same as a Wisconsin dollar. But one extra vote in Wisconsin could swing an election. One extra vote in deep-blue Rhode Island means nothing.

18 | 19 |

Check it out.

20 | 21 |

As the election draws nigh, we'll use artificial intelligence techniques to read these ads (and watch the videos) to summarize what the campaigns are saying to different voters, like if Biden's message focuses on the coronavirus response and Trump's on China. If we can, we'll try to isolate what they're saying to suburban moms versus blue-collar union men and so forth.

22 | 23 | TODO: 24 | 32 | 33 |
34 |

Topics by party

35 | <% @partisanship_topic_proportions.each do |partisanship, topic_proportions| %> 36 | <% next if partisanship == "other" %> 37 |

<%= partisanship %>

38 | 44 | <% end %> 45 |
46 | 47 |
48 |

minimum spend by party

49 | 55 |

we could do per state, or is that too in the weeds?

56 |
57 | 58 | 59 |
60 | <% @grouped.each do |page_id, text_hash_page_wads| %> 61 |

<%= @page_names[page_id] %>

62 | <% text_hash_page_wads.each do |text_hash, wads| %> 63 | <%= render "ads/ad", 64 | page_name: @page_names[page_id], 65 | wad: wads.first, 66 | ad: wads.first.ad, 67 | wads: wads, 68 | states: wads.map{|wad| wad.states}.flatten.uniq %> 69 | <% end %> 70 | <% end %> 71 | 72 |
-------------------------------------------------------------------------------- /config/locales/devise.en.yml: -------------------------------------------------------------------------------- 1 | # Additional translations at https://github.com/plataformatec/devise/wiki/I18n 2 | 3 | en: 4 | devise: 5 | confirmations: 6 | confirmed: "Your email address has been successfully confirmed." 7 | send_instructions: "You will receive an email with instructions for how to confirm your email address in a few minutes." 8 | send_paranoid_instructions: "If your email address exists in our database, you will receive an email with instructions for how to confirm your email address in a few minutes." 9 | failure: 10 | already_authenticated: "You are already signed in." 11 | inactive: "Your account is not activated yet." 12 | invalid: "Invalid %{authentication_keys} or password." 13 | locked: "Your account is locked." 14 | last_attempt: "You have one more attempt before your account is locked." 15 | not_found_in_database: "Invalid %{authentication_keys} or password." 16 | timeout: "Your session expired. Please sign in again to continue." 17 | unauthenticated: "You need to sign in or sign up before continuing." 18 | unconfirmed: "You have to confirm your email address before continuing." 19 | mailer: 20 | confirmation_instructions: 21 | subject: "Confirmation instructions" 22 | reset_password_instructions: 23 | subject: "Reset password instructions" 24 | unlock_instructions: 25 | subject: "Unlock instructions" 26 | email_changed: 27 | subject: "Email Changed" 28 | password_change: 29 | subject: "Password Changed" 30 | omniauth_callbacks: 31 | failure: "Could not authenticate you from %{kind} because \"%{reason}\"." 32 | success: "Successfully authenticated from %{kind} account." 33 | passwords: 34 | no_token: "You can't access this page without coming from a password reset email. If you do come from a password reset email, please make sure you used the full URL provided." 35 | send_instructions: "You will receive an email with instructions on how to reset your password in a few minutes." 36 | send_paranoid_instructions: "If your email address exists in our database, you will receive a password recovery link at your email address in a few minutes." 37 | updated: "Your password has been changed successfully. You are now signed in." 38 | updated_not_active: "Your password has been changed successfully." 39 | registrations: 40 | destroyed: "Bye! Your account has been successfully cancelled. We hope to see you again soon." 41 | signed_up: "Welcome! You have signed up successfully." 42 | signed_up_but_inactive: "You have signed up successfully. However, we could not sign you in because your account is not yet activated." 43 | signed_up_but_locked: "You have signed up successfully. However, we could not sign you in because your account is locked." 44 | signed_up_but_unconfirmed: "A message with a confirmation link has been sent to your email address. Please follow the link to activate your account." 45 | update_needs_confirmation: "You updated your account successfully, but we need to verify your new email address. Please check your email and follow the confirmation link to confirm your new email address." 46 | updated: "Your account has been updated successfully." 47 | updated_but_not_signed_in: "Your account has been updated successfully, but since your password was changed, you need to sign in again" 48 | sessions: 49 | signed_in: "Signed in successfully." 50 | signed_out: "Signed out successfully." 51 | already_signed_out: "Signed out successfully." 52 | unlocks: 53 | send_instructions: "You will receive an email with instructions for how to unlock your account in a few minutes." 54 | send_paranoid_instructions: "If your account exists, you will receive an email with instructions for how to unlock it in a few minutes." 55 | unlocked: "Your account has been unlocked successfully. Please sign in to continue." 56 | errors: 57 | messages: 58 | already_confirmed: "was already confirmed, please try signing in" 59 | confirmation_period_expired: "needs to be confirmed within %{period}, please request a new one" 60 | expired: "has expired, please request a new one" 61 | not_found: "not found" 62 | not_locked: "was not locked" 63 | not_saved: 64 | one: "1 error prohibited this %{resource} from being saved:" 65 | other: "%{count} errors prohibited this %{resource} from being saved:" 66 | -------------------------------------------------------------------------------- /app/views/ads/overview.html.erb: -------------------------------------------------------------------------------- 1 |

Quartz Political Facebook Ads Dashboard

2 | 3 |

We have <%= @ads_count.to_s.reverse.scan(/\d{3}|.+/).join(",").reverse %> ads and <%= @fbpac_ads_count.to_s.reverse.scan(/\d{3}|.+/).join(",").reverse %> from FBPAC.

4 | 5 |
Last updated:
6 | <% if @big_spenders.size > 0 %> 7 |
Big spenders from <%= AdArchiveReport.find(@big_spenders.first.previous_ad_archive_report_id)&.scrape_date %> and <%= @big_spenders.first&.ad_archive_report&.scrape_date %>.
8 | <% end %> 9 | 10 |
11 | 12 |
13 |

Top Advertisers (of all time)

14 | 15 | 16 | <% @top_advertisers.each do |page_id, page_name, amount_spent| %> 17 | 18 | 19 | 20 | 21 | 22 | 23 | <% end %> 24 |
AdvertiserSpend since <%= AdArchiveReport.starting_point.scrape_date.to_date %>
<%= page_name %>$<%= amount_spent.to_s.reverse.scan(/\d{3}|.+/).join(",").reverse %> FB adsadvertiser overview
25 |
26 |
27 |

Top Payers (of all time)

28 | 29 | 30 | <% @top_disclaimers.each do |payer_id, disc, sum| %> 31 | 32 | 33 | 34 | 35 | <% end %> 36 |
PayerSpend since <%= AdArchiveReport::START_DATE %>
<%= disc == disc.upcase ? disc.titleize : disc %>$<%= sum.to_s.reverse.scan(/\d{3}|.+/).join(",").reverse %>FB adspayer overview
37 |
38 | 39 |
40 |

tools that will eventually exist

41 | 45 | 46 |

*New* this week

47 | 52 | 53 |

*New* the past three months

54 | 59 | 60 |

Seen this week

61 | 66 | 67 |

Seen the past three months

68 | 73 | 74 |

Seen since 11/17/2019

75 | 80 | 81 |

Seen ever

82 | 87 | 88 |
89 |
-------------------------------------------------------------------------------- /config/environments/production.rb: -------------------------------------------------------------------------------- 1 | Rails.application.configure do 2 | # Settings specified here will take precedence over those in config/application.rb. 3 | 4 | # Code is not reloaded between requests. 5 | config.cache_classes = true 6 | 7 | # Eager load code on boot. This eager loads most of Rails and 8 | # your application in memory, allowing both threaded web servers 9 | # and those relying on copy on write to perform better. 10 | # Rake tasks automatically ignore this option for performance. 11 | config.eager_load = true 12 | 13 | # Full error reports are disabled and caching is turned on. 14 | config.consider_all_requests_local = false 15 | config.action_controller.perform_caching = true 16 | 17 | # Ensures that a master key has been made available in either ENV["RAILS_MASTER_KEY"] 18 | # or in config/master.key. This key is used to decrypt credentials (and other encrypted files). 19 | # config.require_master_key = true 20 | 21 | # Disable serving static files from the `/public` folder by default since 22 | # Apache or NGINX already handles this. 23 | config.public_file_server.enabled = ENV['RAILS_SERVE_STATIC_FILES'].present? 24 | 25 | # Compress CSS using a preprocessor. 26 | # config.assets.css_compressor = :sass 27 | 28 | # Do not fallback to assets pipeline if a precompiled asset is missed. 29 | config.assets.compile = false 30 | 31 | # Enable serving of images, stylesheets, and JavaScripts from an asset server. 32 | # config.action_controller.asset_host = 'http://assets.example.com' 33 | 34 | # Specifies the header that your server uses for sending files. 35 | # config.action_dispatch.x_sendfile_header = 'X-Sendfile' # for Apache 36 | # config.action_dispatch.x_sendfile_header = 'X-Accel-Redirect' # for NGINX 37 | 38 | # Store uploaded files on the local file system (see config/storage.yml for options). 39 | config.active_storage.service = :local 40 | 41 | # Mount Action Cable outside main process or domain. 42 | # config.action_cable.mount_path = nil 43 | # config.action_cable.url = 'wss://example.com/cable' 44 | # config.action_cable.allowed_request_origins = [ 'http://example.com', /http:\/\/example.*/ ] 45 | 46 | # Force all access to the app over SSL, use Strict-Transport-Security, and use secure cookies. 47 | # config.force_ssl = true 48 | 49 | # Use the lowest log level to ensure availability of diagnostic information 50 | # when problems arise. 51 | config.log_level = :debug 52 | 53 | # Prepend all log lines with the following tags. 54 | config.log_tags = [ :request_id ] 55 | 56 | # Use a different cache store in production. 57 | # config.cache_store = :mem_cache_store 58 | 59 | # Use a real queuing backend for Active Job (and separate queues per environment). 60 | # config.active_job.queue_adapter = :resque 61 | # config.active_job.queue_name_prefix = "AtiDashboard_production" 62 | 63 | config.action_mailer.perform_caching = false 64 | 65 | # Ignore bad email addresses and do not raise email delivery errors. 66 | # Set this to true and configure the email server for immediate delivery to raise delivery errors. 67 | # config.action_mailer.raise_delivery_errors = false 68 | 69 | # Enable locale fallbacks for I18n (makes lookups for any locale fall back to 70 | # the I18n.default_locale when a translation cannot be found). 71 | config.i18n.fallbacks = true 72 | 73 | # Send deprecation notices to registered listeners. 74 | config.active_support.deprecation = :notify 75 | 76 | # Use default logging formatter so that PID and timestamp are not suppressed. 77 | config.log_formatter = ::Logger::Formatter.new 78 | 79 | # Use a different logger for distributed setups. 80 | # require 'syslog/logger' 81 | # config.logger = ActiveSupport::TaggedLogging.new(Syslog::Logger.new 'app-name') 82 | 83 | if ENV["RAILS_LOG_TO_STDOUT"].present? 84 | logger = ActiveSupport::Logger.new(STDOUT) 85 | logger.formatter = config.log_formatter 86 | config.logger = ActiveSupport::TaggedLogging.new(logger) 87 | end 88 | 89 | # Do not dump schema after migrations. 90 | config.active_record.dump_schema_after_migration = false 91 | 92 | # Inserts middleware to perform automatic connection switching. 93 | # The `database_selector` hash is used to pass options to the DatabaseSelector 94 | # middleware. The `delay` is used to determine how long to wait after a write 95 | # to send a subsequent read to the primary. 96 | # 97 | # The `database_resolver` class is used by the middleware to determine which 98 | # database is appropriate to use based on the time delay. 99 | # 100 | # The `database_resolver_context` class is used by the middleware to set 101 | # timestamps for the last write to the primary. The resolver uses the context 102 | # class timestamps to determine how long to wait before reading from the 103 | # replica. 104 | # 105 | # By default Rails will store a last write timestamp in the session. The 106 | # DatabaseSelector middleware is designed as such you can define your own 107 | # strategy for connection switching and pass that into the middleware through 108 | # these configuration options. 109 | # config.active_record.database_selector = { delay: 2.seconds } 110 | # config.active_record.database_resolver = ActiveRecord::Middleware::DatabaseSelector::Resolver 111 | # config.active_record.database_resolver_context = ActiveRecord::Middleware::DatabaseSelector::Resolver::Session 112 | end 113 | -------------------------------------------------------------------------------- /lib/tasks/ad_texts.rake: -------------------------------------------------------------------------------- 1 | require 'digest' 2 | 3 | 4 | # gotta make sure that `clean_text` is the same for ads from both sources. that's the whole point. see below for examples for checking. 5 | 6 | # Ad.where("text ilike '%Charlotte on%'").first.text 7 | # => "Let's get organized! Team Warren will convene a volunteer training in Charlotte on Tuesday, October 1. \n\nJoin us to learn more about how to spread Elizabeth’s vision for big, structural change." 8 | # irb(main):022:0> FbpacAd.where("message ilike '%Charlotte on%'").where(advertiser: "Elizabeth Warren").first.message 9 | # => "

Let's get organized! Team Warren will convene a volunteer training in Charlotte on Tuesday, October 1.

Join us to learn more about how to spread Elizabeth’s vision for big, structural change.

" 10 | # irb(main):023:0> FbpacAd.where("message ilike '%Charlotte on%'").where(advertiser: "Elizabeth Warren").first.clean_text 11 | # => "lets get organized team warren will convene a volunteer training in charlotte on tuesday october 1 join us to learn more about how to spread elizabeths vision for big structural change" 12 | # irb(main):024:0> Ad.where("text ilike '%Charlotte on%'").first.clean_text 13 | # => "lets get organized team warren will convene a volunteer training in charlotte on tuesday october 1 join us to learn more about how to spread elizabeths vision for big structural change" 14 | 15 | 16 | # wilderness project, UNICEF, etc. 17 | BORING_ADVERTISERS = [73970658023, 54684090291, 81517275796, 33110852384, 15239367801, 11131463701] 18 | 19 | 20 | namespace :text do 21 | task clear: :environment do 22 | WritableAd.update_all(text_hash: nil) 23 | end 24 | 25 | task ads: :environment do 26 | start = Time.now 27 | def top_advertiser_page_ids 28 | most_recent_lifelong_report_id = AdArchiveReport.where(kind: 'lifelong', loaded: true).order(:scrape_date).last.id 29 | starting_point_id = AdArchiveReport.starting_point.id 30 | top_advertiser_page_ids = ActiveRecord::Base.connection.exec_query("select start.page_id, start.page_name, current_sum_amount_spent - start_sum_amount_spent from 31 | (SELECT ad_archive_report_pages.page_id as page_id, 32 | ad_archive_report_pages.page_name, 33 | sum(amount_spent) current_sum_amount_spent 34 | FROM ad_archive_report_pages 35 | WHERE ad_archive_report_pages.ad_archive_report_id = #{most_recent_lifelong_report_id} 36 | GROUP BY page_id, page_name) current JOIN (SELECT ad_archive_report_pages.page_id as page_id, 37 | ad_archive_report_pages.page_name, 38 | sum(amount_spent) start_sum_amount_spent 39 | FROM ad_archive_report_pages 40 | WHERE ad_archive_report_pages.ad_archive_report_id = #{starting_point_id} 41 | GROUP BY page_id, page_name) start on start.page_id = current.page_id order by current_sum_amount_spent - start_sum_amount_spent desc limit 10 " 42 | ).rows.map(&:first) 43 | top_advertiser_page_ids - BORING_ADVERTISERS 44 | end 45 | 46 | new_ads = Ad.left_outer_joins(:writable_ad).where(writable_ads: {archive_id: nil}). # ads that don't have a writable ad or whose writable ad doesn't have a text hash in it 47 | # where(page_id: top_advertiser_page_ids) # FOR NOW, limited to the top handful of advertisers 48 | where("ad_creation_time > now() - interval '7 days'") 49 | ads_without_text_hash = WritableAd.where("text_hash is null and archive_id is not null") 50 | 51 | ads_hashed = 0 52 | batch_size = 5000 53 | new_ads.find_in_batches(batch_size: batch_size).map do |batch| 54 | puts "batch (new ads)" 55 | batch.map(&:create_writable_ad!).each do |wad| 56 | wad.ad_text = wad.ad&.create_ad_text!(wad) 57 | wad.save 58 | ads_hashed += 1 59 | end 60 | end 61 | ads_without_text_hash.find_in_batches(batch_size: batch_size).each do |batch| 62 | puts "batch (ads w/o text hash)" 63 | batch.each do |wad| 64 | wad.ad_text = wad.ad.create_ad_text!(wad) 65 | wad.save 66 | ads_hashed += 1 67 | end 68 | end 69 | job = Job.find_by(name: "text:ads") 70 | job_run = job.job_runs.create({ 71 | start_time: start, 72 | end_time: Time.now, 73 | success: true, 74 | }) 75 | 76 | RestClient.post( 77 | ENV["SLACKWH"], 78 | JSON.dump({"text" => "(4/6): text hashing for FB API ads went swimmingly. (#{ads_hashed} ads hashed)" }), 79 | {:content_type => "application/json"} 80 | ) if ads_hashed > 0 && ENV["SLACKWH"] 81 | end 82 | 83 | task fbpac_ads: ["page_ids:fbpac_ads", :environment] do 84 | # eventually this'll be done by the ad catcher, with ATI (but for "collector ads", obvi) 85 | # writable_ad should be created for EVERY new ad. 86 | counter = 0 87 | start = Time.now 88 | 89 | batch_size = 500 90 | FbpacAd.left_outer_joins(:writable_ad).where(writable_ads: {ad_id: nil}).find_in_batches(batch_size: batch_size).each do |new_ads| 91 | counter += 1 92 | puts batch_size * counter 93 | new_ads.each(&:create_writable_ad!) 94 | end 95 | WritableAd.where("text_hash is null and ad_id is not null").find_in_batches(batch_size: batch_size).each do |ads_without_text_hash| 96 | ads_without_text_hash.each do |wad| 97 | wad.text_hash = Digest::SHA1.hexdigest(wad.fbpac_ad.clean_text) 98 | wad.ad_text = create_ad_text!(wad) 99 | wad.save! 100 | end 101 | end 102 | job = Job.find_by(name: "text:fbpac_ads") 103 | job_run = job.job_runs.create({ 104 | start_time: start, 105 | end_time: Time.now, 106 | success: true, 107 | }) 108 | 109 | RestClient.post( 110 | ENV["SLACKWH"], 111 | JSON.dump({"text" => "(3/6): text hashing for collector ads went swimmingly. (#{counter} batches processed)" }), 112 | {:content_type => "application/json"} 113 | ) if counter > 0 && ENV["SLACKWH"] 114 | 115 | 116 | end 117 | end 118 | -------------------------------------------------------------------------------- /app/controllers/youtube_controller.rb: -------------------------------------------------------------------------------- 1 | require 'restclient' 2 | require 'couchrest' 3 | require "base64" 4 | 5 | module CouchRest 6 | class Database 7 | def find(query) 8 | connection.post "#{path}/_find", query 9 | end 10 | 11 | end 12 | end 13 | 14 | 15 | class YoutubeController < ApplicationController 16 | 17 | before_action :force_trailing_slash 18 | 19 | REASONS_KINDS = [ 20 | "This ad is based on:", 21 | "This ad may be based on:", 22 | "You've turned off ad personalization from Google, so this ad is not customized based on your data. This ad was shown based on other factors, for example:", 23 | "Motifs de sélection de cette annonce :" 24 | ] 25 | 26 | @@server = CouchRest.new(ENV["DBURL"]) 27 | @@ads_db = @@server.database("youtubeads") 28 | @@recs_db = @@server.database("youtuberecs") 29 | 30 | 31 | def group_ad_fragments(raw_ad_fragments) 32 | grouped_ads = raw_ad_fragments.group_by{|ad| [(ad["host"] || {})["url"], ad["user"], ad["ad"]['advertiser']&.split(" ")&.last&.downcase&.gsub("www.", '')] } # group ads that were seen by the same person on the same site and which reference the same advertiser 33 | ads = grouped_ads.map do |host_user_advertiser, ungrouped_ads| # then merge the resulting ad objects into a single merged-ad object 34 | ads_grouped_by_reasons = ungrouped_ads.group_by{|ad| ad["ad"]["reasons"] } 35 | ads_grouped_by_reasons.delete(nil) # ignore ads with no reasons because they are all just type=adActionInterstitialRenderer, with advertiser a duplicate of another ad (of a different type) that's present. so learn nothing other than that there was an adActionInterstitialRenderer used at some point which I think I don't care about. 36 | merged_ads = ads_grouped_by_reasons.map do |reason, ads| 37 | types = ungrouped_ads.map{|a| a["ad"]["type"]} # preserving type 38 | ad = ungrouped_ads.reduce(&:merge) 39 | ad["ad"]["type"] = types 40 | ad 41 | end 42 | merged_ads 43 | end.flatten 44 | ads 45 | end 46 | 47 | def index 48 | # raw_ads = @@ads_db.all_docs(include_docs: true)["rows"].map{|a| a["doc"]} 49 | # grouped_ads = raw_ads.group_by{|ad| [(ad["host"] || {})["url"], ad["user"], ad["ad"]['advertiser']&.split(" ")&.last&.downcase&.gsub("www.", '')] } # group ads that were seen by the same person on the same site and which reference the same advertiser 50 | # ads = grouped_ads.map do |host_user_advertiser, ungrouped_ads| # then merge the resulting ad objects into a single merged-ad object 51 | # ads_grouped_by_reasons = ungrouped_ads.group_by{|ad| ad["ad"]["reasons"] } 52 | # ads_grouped_by_reasons.delete(nil) # ignore ads with no reasons because they are all just type=adActionInterstitialRenderer, with advertiser a duplicate of another ad (of a different type) that's present. so learn nothing other than that there was an adActionInterstitialRenderer used at some point which I think I don't care about. 53 | # merged_ads = ads_grouped_by_reasons.map do |reason, ads| 54 | # types = ungrouped_ads.map{|a| a["ad"]["type"]} # preserving type 55 | # ad = ungrouped_ads.reduce(&:merge) 56 | # ad["ad"]["type"] = types 57 | # ad 58 | # end 59 | # merged_ads 60 | # end.flatten 61 | ad_fragments = @@ads_db.all_docs(include_docs: true)["rows"].map{|a| a["doc"]} 62 | ads = group_ad_fragments(ad_fragments) 63 | @ads_count = ads.count 64 | 65 | political_ads = @@ads_db.find({"selector" => 66 | { "$nor" => REASONS_KINDS.map{|kind| { "ad.reasons_title" => kind }} } 67 | })["docs"] 68 | @political_ads_count = political_ads.size 69 | 70 | @political_advertisers = political_ads.group_by{|ad| ad["ad"]["reasons_title"].split("\r\n\r\n")[0] } # + " " + ad["ad"]["advertiser"].to_s (usually "advertiser" is actually a URL) 71 | @political_targetings = political_ads.map{|ad| ad["ad"]["reasons"] }.flatten.group_by{|reason| reason }.map{|reason, items| [reason, items.size]} 72 | 73 | @advertisers = ads.group_by{|ad| ad["ad"]["advertiser"] }.reject{|a, b| a.nil? || b.size == 1} 74 | @targetings = ads.map{|ad| ad["ad"]["reasons"] }.flatten.group_by{|reason| reason }.map{|reason, items| [reason, items.size]}.reject{|a, b| a.nil? } 75 | render template: "youtube/index" 76 | end 77 | 78 | def advertiser 79 | targ = Base64.urlsafe_decode64(params['targ']) 80 | raw_ad_fragments = @@ads_db.find({"selector" => 81 | {"$or" => REASONS_KINDS.map{|kind| 82 | { "ad.reasons_title" => targ + "\r\n\r\n" + kind } 83 | }} 84 | })["docs"] 85 | @matching_ads = group_ad_fragments(raw_ad_fragments) 86 | puts @matching_ads.inspect 87 | @query = "Reason: #{targ}" 88 | render template: "youtube/list" 89 | end 90 | 91 | def targeting 92 | targ = Base64.urlsafe_decode64(params['targ']) 93 | raw_ad_fragments = @@ads_db.find({"selector" => 94 | {"$and" => [ 95 | { "$nor" => REASONS_KINDS.map{|kind| { "ad.reasons_title" => kind }} }, 96 | { "ad.reasons" => { 97 | "$elemMatch" => { 98 | "$eq" => targ 99 | } 100 | }} 101 | ]} 102 | })["docs"] 103 | @matching_ads = group_ad_fragments(raw_ad_fragments) 104 | @query = "Reason: #{targ}" 105 | render template: "youtube/list" 106 | end 107 | 108 | def targeting_all 109 | targ = Base64.urlsafe_decode64(params['targ']) 110 | raw_ad_fragments = @@ads_db.find({"selector" => 111 | { "ad.reasons" => { 112 | "$elemMatch" => { 113 | "$eq" => targ 114 | } 115 | }} 116 | })["docs"] 117 | @matching_ads = group_ad_fragments(raw_ad_fragments) 118 | @query = "Reason: #{targ}" 119 | render template: "youtube/list" 120 | end 121 | 122 | def advertiser_all 123 | targ = Base64.urlsafe_decode64(params['targ']) 124 | raw_ad_fragments = @@ads_db.find({"selector" => 125 | {"$or" => REASONS_KINDS.map{|kind| 126 | { "ad.advertiser" => targ } 127 | }} 128 | })["docs"] 129 | @matching_ads = group_ad_fragments(raw_ad_fragments) 130 | @query = "Reason: #{targ}" 131 | render template: "youtube/list" 132 | end 133 | 134 | 135 | 136 | end 137 | -------------------------------------------------------------------------------- /Gemfile.lock: -------------------------------------------------------------------------------- 1 | GIT 2 | remote: https://github.com/elastic/elasticsearch-rails.git 3 | revision: 7bbe7ec89e9a31380e4a8f0bff886559d80dc27c 4 | branch: 6.x 5 | specs: 6 | elasticsearch-model (6.1.0) 7 | activesupport (> 3) 8 | elasticsearch (> 1) 9 | hashie 10 | elasticsearch-rails (6.1.0) 11 | 12 | GIT 13 | remote: https://github.com/elastic/elasticsearch-ruby.git 14 | revision: bd78944f3ef3abc37defc588325cb872353ba6c3 15 | branch: 6.x 16 | specs: 17 | elasticsearch-dsl (0.1.5) 18 | 19 | GEM 20 | remote: https://rubygems.org/ 21 | specs: 22 | actioncable (6.0.0) 23 | actionpack (= 6.0.0) 24 | nio4r (~> 2.0) 25 | websocket-driver (>= 0.6.1) 26 | actionmailbox (6.0.0) 27 | actionpack (= 6.0.0) 28 | activejob (= 6.0.0) 29 | activerecord (= 6.0.0) 30 | activestorage (= 6.0.0) 31 | activesupport (= 6.0.0) 32 | mail (>= 2.7.1) 33 | actionmailer (6.0.0) 34 | actionpack (= 6.0.0) 35 | actionview (= 6.0.0) 36 | activejob (= 6.0.0) 37 | mail (~> 2.5, >= 2.5.4) 38 | rails-dom-testing (~> 2.0) 39 | actionpack (6.0.0) 40 | actionview (= 6.0.0) 41 | activesupport (= 6.0.0) 42 | rack (~> 2.0) 43 | rack-test (>= 0.6.3) 44 | rails-dom-testing (~> 2.0) 45 | rails-html-sanitizer (~> 1.0, >= 1.2.0) 46 | actionpack-action_caching (1.2.1) 47 | actionpack (>= 4.0.0) 48 | actiontext (6.0.0) 49 | actionpack (= 6.0.0) 50 | activerecord (= 6.0.0) 51 | activestorage (= 6.0.0) 52 | activesupport (= 6.0.0) 53 | nokogiri (>= 1.8.5) 54 | actionview (6.0.0) 55 | activesupport (= 6.0.0) 56 | builder (~> 3.1) 57 | erubi (~> 1.4) 58 | rails-dom-testing (~> 2.0) 59 | rails-html-sanitizer (~> 1.1, >= 1.2.0) 60 | active_record_extended (1.4.0) 61 | activerecord (>= 5.0, < 6.1) 62 | ar_outer_joins (~> 0.2) 63 | pg (< 2.0) 64 | activejob (6.0.0) 65 | activesupport (= 6.0.0) 66 | globalid (>= 0.3.6) 67 | activemodel (6.0.0) 68 | activesupport (= 6.0.0) 69 | activerecord (6.0.0) 70 | activemodel (= 6.0.0) 71 | activesupport (= 6.0.0) 72 | activestorage (6.0.0) 73 | actionpack (= 6.0.0) 74 | activejob (= 6.0.0) 75 | activerecord (= 6.0.0) 76 | marcel (~> 0.3.1) 77 | activesupport (6.0.0) 78 | concurrent-ruby (~> 1.0, >= 1.0.2) 79 | i18n (>= 0.7, < 2) 80 | minitest (~> 5.1) 81 | tzinfo (~> 1.1) 82 | zeitwerk (~> 2.1, >= 2.1.8) 83 | addressable (2.7.0) 84 | public_suffix (>= 2.0.2, < 5.0) 85 | ar_outer_joins (0.2.0) 86 | activerecord (>= 3.2) 87 | aws-eventstream (1.1.0) 88 | aws-partitions (1.302.0) 89 | aws-sdk-core (3.94.0) 90 | aws-eventstream (~> 1, >= 1.0.2) 91 | aws-partitions (~> 1, >= 1.239.0) 92 | aws-sigv4 (~> 1.1) 93 | jmespath (~> 1.0) 94 | aws-sdk-kms (1.30.0) 95 | aws-sdk-core (~> 3, >= 3.71.0) 96 | aws-sigv4 (~> 1.1) 97 | aws-sdk-s3 (1.63.0) 98 | aws-sdk-core (~> 3, >= 3.83.0) 99 | aws-sdk-kms (~> 1) 100 | aws-sigv4 (~> 1.1) 101 | aws-sigv4 (1.1.2) 102 | aws-eventstream (~> 1.0, >= 1.0.2) 103 | bcrypt (3.1.13) 104 | bindex (0.8.1) 105 | bootsnap (1.4.5) 106 | msgpack (~> 1.0) 107 | builder (3.2.3) 108 | byebug (11.0.1) 109 | capybara (3.29.0) 110 | addressable 111 | mini_mime (>= 0.1.3) 112 | nokogiri (~> 1.8) 113 | rack (>= 1.6.0) 114 | rack-test (>= 0.6.3) 115 | regexp_parser (~> 1.5) 116 | xpath (~> 3.2) 117 | childprocess (2.0.0) 118 | rake (< 13.0) 119 | concurrent-ruby (1.1.5) 120 | couchrest (2.0.1) 121 | httpclient (~> 2.8) 122 | mime-types (>= 1.15) 123 | multi_json (~> 1.7) 124 | crass (1.0.4) 125 | devise (4.7.1) 126 | bcrypt (~> 3.0) 127 | orm_adapter (~> 0.1) 128 | railties (>= 4.1.0) 129 | responders 130 | warden (~> 1.2.3) 131 | domain_name (0.5.20190701) 132 | unf (>= 0.0.5, < 1.0.0) 133 | elasticsearch (7.3.0) 134 | elasticsearch-api (= 7.3.0) 135 | elasticsearch-transport (= 7.3.0) 136 | elasticsearch-api (7.3.0) 137 | multi_json 138 | elasticsearch-transport (7.3.0) 139 | faraday 140 | multi_json 141 | erubi (1.9.0) 142 | faraday (0.17.0) 143 | multipart-post (>= 1.2, < 3) 144 | ffi (1.11.1) 145 | globalid (0.4.2) 146 | activesupport (>= 4.2.0) 147 | hashie (3.6.0) 148 | http-accept (1.7.0) 149 | http-cookie (1.0.3) 150 | domain_name (~> 0.5) 151 | httpclient (2.8.3) 152 | i18n (1.6.0) 153 | concurrent-ruby (~> 1.0) 154 | jar_wrapper (0.1.8) 155 | zip 156 | jbuilder (2.9.1) 157 | activesupport (>= 4.2.0) 158 | jmespath (1.4.0) 159 | listen (3.1.5) 160 | rb-fsevent (~> 0.9, >= 0.9.4) 161 | rb-inotify (~> 0.9, >= 0.9.7) 162 | ruby_dep (~> 1.2) 163 | loofah (2.3.0) 164 | crass (~> 1.0.2) 165 | nokogiri (>= 1.5.9) 166 | mail (2.7.1) 167 | mini_mime (>= 0.1.1) 168 | marcel (0.3.3) 169 | mimemagic (~> 0.3.2) 170 | method_source (0.9.2) 171 | mime-types (3.3) 172 | mime-types-data (~> 3.2015) 173 | mime-types-data (3.2019.1009) 174 | mimemagic (0.3.3) 175 | mini_mime (1.0.2) 176 | mini_portile2 (2.4.0) 177 | minitest (5.12.2) 178 | msgpack (1.3.1) 179 | multi_json (1.14.1) 180 | multipart-post (2.1.1) 181 | netrc (0.11.0) 182 | nio4r (2.5.2) 183 | nokogiri (1.10.4) 184 | mini_portile2 (~> 2.4.0) 185 | orm_adapter (0.5.0) 186 | pg (1.1.4) 187 | pg_search (2.3.2) 188 | activerecord (>= 5.2) 189 | activesupport (>= 5.2) 190 | public_suffix (4.0.1) 191 | puma (3.12.1) 192 | rack (2.0.7) 193 | rack-cors (1.1.1) 194 | rack (>= 2.0.0) 195 | rack-proxy (0.6.5) 196 | rack 197 | rack-test (1.1.0) 198 | rack (>= 1.0, < 3) 199 | rails (6.0.0) 200 | actioncable (= 6.0.0) 201 | actionmailbox (= 6.0.0) 202 | actionmailer (= 6.0.0) 203 | actionpack (= 6.0.0) 204 | actiontext (= 6.0.0) 205 | actionview (= 6.0.0) 206 | activejob (= 6.0.0) 207 | activemodel (= 6.0.0) 208 | activerecord (= 6.0.0) 209 | activestorage (= 6.0.0) 210 | activesupport (= 6.0.0) 211 | bundler (>= 1.3.0) 212 | railties (= 6.0.0) 213 | sprockets-rails (>= 2.0.0) 214 | rails-dom-testing (2.0.3) 215 | activesupport (>= 4.2.0) 216 | nokogiri (>= 1.6) 217 | rails-html-sanitizer (1.2.0) 218 | loofah (~> 2.2, >= 2.2.2) 219 | railties (6.0.0) 220 | actionpack (= 6.0.0) 221 | activesupport (= 6.0.0) 222 | method_source 223 | rake (>= 0.8.7) 224 | thor (>= 0.20.3, < 2.0) 225 | rake (12.3.3) 226 | rb-fsevent (0.10.3) 227 | rb-inotify (0.10.0) 228 | ffi (~> 1.0) 229 | regexp_parser (1.6.0) 230 | responders (3.0.0) 231 | actionpack (>= 5.0) 232 | railties (>= 5.0) 233 | rest-client (2.1.0) 234 | http-accept (>= 1.7.0, < 2.0) 235 | http-cookie (>= 1.0.2, < 2.0) 236 | mime-types (>= 1.16, < 4.0) 237 | netrc (~> 0.8) 238 | ruby-progressbar (1.10.1) 239 | ruby_dep (1.5.0) 240 | rubyzip (1.3.0) 241 | sassc (2.2.1) 242 | ffi (~> 1.9) 243 | sassc-rails (2.1.2) 244 | railties (>= 4.0.0) 245 | sassc (>= 2.0) 246 | sprockets (> 3.0) 247 | sprockets-rails 248 | tilt 249 | selenium (0.2.11) 250 | jar_wrapper 251 | selenium-webdriver (3.142.5) 252 | childprocess (>= 0.5, < 3.0) 253 | rubyzip (>= 1.2.2) 254 | spring (2.1.0) 255 | spring-watcher-listen (2.0.1) 256 | listen (>= 2.7, < 4.0) 257 | spring (>= 1.2, < 3.0) 258 | sprockets (3.7.2) 259 | concurrent-ruby (~> 1.0) 260 | rack (> 1, < 3) 261 | sprockets-rails (3.2.1) 262 | actionpack (>= 4.0) 263 | activesupport (>= 4.0) 264 | sprockets (>= 3.0.0) 265 | sqlite3 (1.4.1) 266 | thor (0.20.3) 267 | thread_safe (0.3.6) 268 | tilt (2.0.10) 269 | turbolinks (5.2.1) 270 | turbolinks-source (~> 5.2) 271 | turbolinks-source (5.2.0) 272 | tzinfo (1.2.5) 273 | thread_safe (~> 0.1) 274 | unf (0.1.4) 275 | unf_ext 276 | unf_ext (0.0.7.6) 277 | warden (1.2.8) 278 | rack (>= 2.0.6) 279 | web-console (4.0.1) 280 | actionview (>= 6.0.0) 281 | activemodel (>= 6.0.0) 282 | bindex (>= 0.4.0) 283 | railties (>= 6.0.0) 284 | webdrivers (4.1.2) 285 | nokogiri (~> 1.6) 286 | rubyzip (~> 1.0) 287 | selenium-webdriver (>= 3.0, < 4.0) 288 | webpacker (4.0.7) 289 | activesupport (>= 4.2) 290 | rack-proxy (>= 0.6.1) 291 | railties (>= 4.2) 292 | websocket-driver (0.7.1) 293 | websocket-extensions (>= 0.1.0) 294 | websocket-extensions (0.1.4) 295 | will_paginate (3.1.8) 296 | xpath (3.2.0) 297 | nokogiri (~> 1.8) 298 | zeitwerk (2.1.10) 299 | zip (2.0.2) 300 | 301 | PLATFORMS 302 | ruby 303 | 304 | DEPENDENCIES 305 | actionpack-action_caching 306 | active_record_extended 307 | aws-sdk-s3 (~> 1) 308 | bootsnap (>= 1.4.2) 309 | byebug 310 | capybara (>= 2.15) 311 | couchrest 312 | devise 313 | elasticsearch-dsl! 314 | elasticsearch-model! 315 | elasticsearch-rails! 316 | jbuilder (~> 2.7) 317 | listen (>= 3.0.5, < 3.2) 318 | pg 319 | pg_search 320 | puma (~> 3.11) 321 | rack-cors 322 | rails (~> 6.0.0) 323 | rest-client 324 | ruby-progressbar 325 | sassc-rails 326 | selenium 327 | selenium-webdriver 328 | spring 329 | spring-watcher-listen (~> 2.0.0) 330 | sqlite3 (~> 1.4) 331 | turbolinks (~> 5) 332 | tzinfo-data 333 | web-console (>= 3.3.0) 334 | webdrivers 335 | webpacker (~> 4.0) 336 | will_paginate (~> 3.1.0) 337 | 338 | RUBY VERSION 339 | ruby 2.6.5p114 340 | 341 | BUNDLED WITH 342 | 1.17.2 343 | -------------------------------------------------------------------------------- /core_advertisers_20200422.csv: -------------------------------------------------------------------------------- 1 | '202626512363,U.S. Census Bureau,10689550,FALSE, 2 | '925007000949719,ExxonMobil,2260566,FALSE, 3 | '54684090291,Partnership for Drug-Free Kids,2192929,FALSE, 4 | '11131463701,truth,1948561,FALSE, 5 | '33110852384,Doctors Without Borders/ Médecins Sans Frontières (MSF),1944125,FALSE, 6 | '121388362246,Goldman Sachs,1677662,FALSE, 7 | '172758932738142,PhRMA,1573799,FALSE, 8 | '328072970642479,Jaime Harrison,1223859,FALSE,dem 9 | '38235851310,USA for UNHCR,1014487,FALSE,dem 10 | '111393882812494,Amy McGrath,984344,FALSE,dem 11 | '2352630938293825,Mytacticalpromos,971347,FALSE, 12 | '6204742571,Sierra Club,951583,FALSE, 13 | '6985452607,Brita USA,828296,FALSE, 14 | '25852730092,Care2,828114,FALSE, 15 | '109092142462587,Adam Schiff,810507,FALSE, 16 | '81517275796,UNICEF USA,807026,FALSE, 17 | '117580534956446,Kellogg's Frosted Flakes,806417,FALSE, 18 | '163148530407080,Captain Mark Kelly,795795,FALSE, 19 | '2330577043935831,Sara Gideon,737857,FALSE, 20 | '30139072158,CARE,734635,FALSE, 21 | '7976226799,The Daily Show,723238,FALSE, 22 | '1194865807192273,Conservative Gear,715719,FALSE, 23 | '717115395103069,PrintedKicks,670240,FALSE, 24 | '681626158940626,Mission Roll Call,630327,FALSE, 25 | '159616034235,Walmart,613976,FALSE, 26 | '5435784683,Greenpeace USA,593519,FALSE, 27 | '246323280108,American Medical Association (AMA),593365,FALSE, 28 | '136299005939,Heifer International,524606,FALSE, 29 | '157546398049625,Marsy's Law for Wisconsin,522424,FALSE, 30 | '216397232833,Penzeys Spices,507657,FALSE, 31 | '11791104453,NRDC,492143,FALSE, 32 | '203197030603239,8 Billion Trees,486746,FALSE, 33 | '19192438096,AAAS - The American Association for the Advancement of Science,473839,FALSE, 34 | '132492254050920,American AF,473782,FALSE, 35 | '15239367801,The Wilderness Society,468761,FALSE, 36 | '281844617168,Jewish Voice Ministries International,453852,FALSE, 37 | '206623622690681,NYC Department of Health and Mental Hygiene,453366,FALSE, 38 | '15687409793,World Wildlife Fund,436055,FALSE, 39 | '56680318666,Illinois Policy,432968,FALSE, 40 | '897994503631860,HealthInsurance.net,430785,FALSE, 41 | '35169196082,Project HOPE,429935,FALSE, 42 | '174009339411865,SmartNews,425446,FALSE, 43 | '179204028758675,California Secretary of State,423912,FALSE, 44 | '8532248414,Wounded Warrior Project,414253,FALSE, 45 | '109784800371914,Protect Kids: Fight Flavored E-Cigarettes,401960,FALSE, 46 | '1456310534453054,Veteran Loans Online,396335,FALSE, 47 | '97493741436,John W. Hickenlooper,394903,FALSE, 48 | '111428049989,Cal Cunningham,386592,FALSE, 49 | '2353537341582545,Unicorn Politics,361826,FALSE, 50 | '530604487365010,Old Glory Flagpole Kits,360208,FALSE, 51 | '169294473121454,Policetees.com,349598,FALSE, 52 | '5720973755,Defenders of Wildlife,344849,FALSE, 53 | '113697513378680,Clean Energy Matters,336227,FALSE, 54 | '121626482110,Operation Smile,335334,FALSE, 55 | '950998258312654,Knowhere,327256,FALSE, 56 | '151485624911761,NYC Mayor's Office,326892,FALSE, 57 | '23875187730,Feeding America,322070,FALSE, 58 | '1374134752633656,New Health Plans,318871,FALSE, 59 | '8047221596,Save the Children US,316910,FALSE, 60 | '106700630683805,Bill Conway,316193,FALSE, 61 | '1449461518652183,4Patriots,315543,FALSE, 62 | '8325302025,Friends of the Earth U.S.,315357,FALSE, 63 | '15041226930,Oxfam,314454,FALSE, 64 | '105092502914147,SoFi,314142,FALSE, 65 | '140499052681592,Bank of the West,313619,FALSE, 66 | '17911165812,Covenant House International,307361,FALSE, 67 | '167586229923327,World Food Program USA,307043,FALSE, 68 | '293225127381200,Patriot Depot,304245,FALSE, 69 | '658739734196897,CVS Health,302277,FALSE, 70 | '21559454473,FAIR,301027,FALSE, 71 | '1852520761657982,4ocean,300057,FALSE, 72 | '1690071827911682,Cleancult,299920,FALSE, 73 | '6756153498,Mike Bloomberg,62221879,TRUE,dem 74 | '153080620724,Donald J. Trump,13080628,TRUE,gop 75 | '416707608450706,Tom Steyer,10041118,TRUE,dem 76 | '7860876103,Joe Biden,8994072,TRUE,dem 77 | '124955570892789,Bernie Sanders,8543893,TRUE,dem 78 | '156968474833352,Stop Republicans,5394386,TRUE,dem 79 | '38471053686,Elizabeth Warren,3741988,TRUE,dem 80 | '1039701332716228,Pete Buttigieg,2748411,TRUE,dem 81 | '6726182861,Mike Pence,1783361,TRUE,gop 82 | '20787991568,Senate Democrats,1660588,TRUE,dem 83 | '96935476345,Democratic Governors Association (DGA),1367695,TRUE,dem 84 | '107858693983382,The GOP Shop,1350609,TRUE,gop 85 | '127225910653607,PragerU,1267348,TRUE,gop 86 | '92925746942,Judicial Watch,1177199,TRUE,gop 87 | '8934429638,Planned Parenthood Action,1171303,TRUE,dem 88 | '425408274980132,Let's Be Honest,1091988,TRUE,dem 89 | '791543660926838,"I'll go ahead and keep my guns, Thanks",1091056,TRUE,gop 90 | '259130650776119,Mitch McConnell,1024129,TRUE,gop 91 | '1884107931849913,Republicans for the Rule of Law,994750,TRUE,dem 92 | '127755494595870,Progressive Takeover,942296,TRUE,dem 93 | '1316372698453411,Alexandria Ocasio-Cortez,912770,TRUE,dem 94 | '112041183486666,Project Sunshine,909522,TRUE,dem 95 | '156994304362574,American Bridge,907710,TRUE,dem 96 | '602714436529710,End Citizens United,890721,TRUE,dem 97 | '1615018905412860,Progressive Turnout Project,841378,TRUE,dem 98 | '158791100909689,Concerned Veterans for America,808825,TRUE,gop 99 | '9324910069,NRSC,782747,TRUE,gop 100 | '55549065733,Voto Latino,760032,TRUE,dem 101 | '266790296879,Bloomberg,743885,TRUE,dem 102 | '342294162453301,Planned Parenthood,725908,TRUE,dem 103 | '110749446963570,Courier,716744,TRUE,dem 104 | '665345483483486,NRA - National Rifle Association of America,706217,TRUE,gop 105 | '309798029121030,Everytown for Gun Safety,704839,TRUE,dem 106 | '1711465445765878,Nancy Pelosi,684413,TRUE,dem 107 | '80562389320,NARAL Pro-Choice America,681454,TRUE,dem 108 | '228754857151053,House Majority PAC,679477,TRUE,dem 109 | '260754507296508,The Voter Participation Center,659981,TRUE,dem 110 | '196382377057315,Learn Our History,659404,TRUE,gop 111 | '7606381190,Amy Klobuchar,657279,TRUE,dem 112 | '165987503528599,Lindsey Graham,652803,TRUE,gop 113 | '101617224555221,Nancy Pelosi Elects Democrats,644577,TRUE,dem 114 | '345504286342057,Keeping America Great Again,643721,TRUE,gop 115 | '309308412520731,Sandy Hook Promise,609007,TRUE,dem 116 | '242077219243971,The Presidential Coalition,597243,TRUE,gop 117 | '562149327457702,Andrew Yang for President 2020,582094,TRUE,dem 118 | '830121337093232,National Democratic Training Committee,503667,TRUE,dem 119 | '332716170522025,America First Action,496810,TRUE,gop 120 | '159964696102,Kevin McCarthy,494167,TRUE,gop 121 | '12301006942,Democratic Party,489355,TRUE,dem 122 | '2283535575207809,Ditch Mitch,470554,TRUE,dem 123 | '261663760956534,America First Policies,470232,TRUE,gop 124 | '174866249236469,Tulsi Gabbard,459070,TRUE,dem 125 | '560140464159364,Stand for the 2nd Amendment,457892,TRUE,gop 126 | '85452072376,Newsmax,447059,TRUE,gop 127 | '383877235817031,Keep America Great,447055,TRUE,gop 128 | '7292655492,MoveOn,425749,TRUE,dem 129 | '18982436812,ACLU,424346,TRUE,dem 130 | '239452426714,American Action Network,417119,TRUE,gop 131 | '1155212867963791,AFP Action,409424,TRUE,gop 132 | '139599847972,National Nurses United,399632,TRUE,dem 133 | '197760674021143,Indivisible Guide,399523,TRUE,dem 134 | '67945990059,The American Petroleum Institute,377733,TRUE,gop 135 | '605168822998856,314 Action,373884,TRUE,dem 136 | '592347634460462,Partnership for America's Health Care Future,365738,TRUE,other 137 | '1338822066131069,Women's March,359387,TRUE,dem 138 | '305832706292882,American Gun Association,345356,TRUE,gop 139 | '102711087850827,Hablemos Claro USA,345058,TRUE,dem 140 | '24330467048,Americans for Prosperity,339326,TRUE,gop 141 | '21375324480,The Heritage Foundation,328845,TRUE,gop 142 | '143055773143383,Patients For Affordable Drugs NOW,327960,TRUE,other 143 | '16477459734,League of Conservation Voters,327753,TRUE,dem 144 | '209516845093,NRDC Action Fund,327401,TRUE,dem 145 | '203805062990264,Ben Shapiro,326919,TRUE,gop 146 | '123192635089,GOP,319878,TRUE,gop 147 | '32682889528,U.S. Chamber of Commerce,305188,TRUE,other 148 | '213792049451918,Stand For America Now,302753,TRUE,gop 149 | '571182609571332,NextGen America,300251,TRUE,dem 150 | '1025055787535714,A Case for Women,1036707,, 151 | '551328948361758,For Our Future WI,299251,, 152 | '2602473023098666,Democratic Polling Center,298251,, 153 | '393932827634250,Dissent Pins,297328,, 154 | '46093654473,National Republican Congressional Committee,296909,TRUE,gop 155 | '6500552187,Hulu,293514,, 156 | '7483836629,Ancestry,291922,, 157 | '123450447708791,Energy Upgrade California,289041,, 158 | '1010143499143169,Medicare Savings,288771,, 159 | '58433611572,Seventh Generation,288550,, 160 | '7263476274,Mercy Corps,288082,, 161 | '1596671280578981,Schools & Communities First,283914,, 162 | '314499212242,Pfizer,275550,, 163 | '95356211637,Ben Ferguson,269684,, 164 | '2267405033373814,Doctor Patient Unity,267020,, 165 | '944580302249505,NYC Human Resources Administration,265109,, 166 | '57259033959,AARP,265002,, 167 | '119746614730099,Reliant Energy ®,262888,, 168 | '142992759853421,Be A Hero,262831,, 169 | '118502801658228,Young Kim,256021,, 170 | '277005566267077,Blueland,248645,, 171 | '534897536611506,Nuclear Powers IL,248360,, 172 | '834645469890446,Devin Nunes,247583,, 173 | '345378512258813,Elise Stefanik,247422,, 174 | '1145019702185006,Alliance for Safety and Justice,246948,, 175 | '216235145056069,Stop The HIT,243547,, 176 | '18813753280,Human Rights Campaign,242121,, 177 | '8501031021,VoteVets.org,241259,, 178 | '148060455834119,YES For a Better San Diego,240349,, 179 | '1691667177798461,Brut,240281,, 180 | '344690446467832,The Copper Courier,239129,, 181 | '104006944272732,Honest Arizona,236307,, 182 | '237477017606,Club for Growth,234482,TRUE,gop 183 | '1109093812491437,Blaze Media,232611,TRUE,gop 184 | '72931140459,Center for Biological Diversity,230434,, 185 | '6713653788,Gary Peters,227874,, 186 | '137569740231452,Respect The Look,226078,, 187 | '321250352104571,Valerie Plame,225635,, 188 | '120551431317,Nickelodeon,225011,, 189 | '21492729544,Catholic Relief Services,224121,, 190 | '173837496125511,Conscious Step,224052,, 191 | '294196590710262,FWD.us,221938,, 192 | '1502654549792632,Protect Our Care,221710,, 193 | '152065518140304,Wildlife Conservation Society,217543,, 194 | '461204744067686,Family Protection Association,216520,, 195 | '49651563727,Live Action,215581,, 196 | '1479923688811438,Geothermal,213936,, 197 | '332669333486567,California Republican Party,211801,, 198 | '18479402361,No Kid Hungry,208148,, 199 | '104745474214057,HRA - OTC Birth Control,207133,, 200 | '114501575391943,Compare Medicare Quotes,205849,, 201 | '105756057589080,Lincoln Project,,TRUE,dem -------------------------------------------------------------------------------- /db/schema.rb: -------------------------------------------------------------------------------- 1 | # This file is auto-generated from the current state of the database. Instead 2 | # of editing this file, please use the migrations feature of Active Record to 3 | # incrementally modify your database, and then regenerate this schema definition. 4 | # 5 | # This file is the source Rails uses to define your schema when running `rails 6 | # db:schema:load`. When creating a new database, `rails db:schema:load` tends to 7 | # be faster and is potentially less error prone than running all of your 8 | # migrations from scratch. Old migrations may fail to apply correctly if those 9 | # migrations use external dependencies or application code. 10 | # 11 | # It's strongly recommended that you check this file into your version control system. 12 | 13 | ActiveRecord::Schema.define(version: 2020_05_08_174349) do 14 | 15 | # These are extensions that must be enabled in order to support this database 16 | enable_extension "plpgsql" 17 | enable_extension "postgres_fdw" 18 | enable_extension "unaccent" 19 | 20 | create_table "ad_archive_report_pages", force: :cascade do |t| 21 | t.bigint "page_id" 22 | t.string "page_name" 23 | t.string "disclaimer" 24 | t.integer "amount_spent" 25 | t.integer "ads_count" 26 | t.integer "ad_archive_report_id" 27 | t.datetime "created_at", precision: 6, null: false 28 | t.datetime "updated_at", precision: 6, null: false 29 | t.integer "ads_this_tranche" 30 | t.integer "spend_this_tranche" 31 | t.integer "amount_spent_since_start_date" 32 | t.index ["ad_archive_report_id", "page_id", "disclaimer"], name: "index_aarps_aar_id_page_id_discl" 33 | t.index ["ad_archive_report_id", "page_id"], name: "index_ad_archive_report_pages_on_ad_archive_report_id_page_id" 34 | end 35 | 36 | create_table "ad_archive_reports", force: :cascade do |t| 37 | t.datetime "scrape_date" 38 | t.text "s3_url" 39 | t.text "kind" 40 | t.boolean "loaded", default: false 41 | t.datetime "created_at", precision: 6, null: false 42 | t.datetime "updated_at", precision: 6, null: false 43 | end 44 | 45 | create_table "ad_texts", force: :cascade do |t| 46 | t.text "text" 47 | t.string "text_hash" 48 | t.text "vec" 49 | t.datetime "created_at", precision: 6, null: false 50 | t.datetime "updated_at", precision: 6, null: false 51 | t.text "search_text" 52 | t.tsvector "tsv" 53 | t.bigint "page_id" 54 | t.text "advertiser" 55 | t.text "paid_for_by" 56 | t.datetime "first_seen" 57 | t.datetime "last_seen" 58 | t.index ["advertiser"], name: "index_ad_texts_on_advertiser" 59 | t.index ["first_seen", "text_hash"], name: "index_ad_texts_on_first_seen_and_text_hash", order: { first_seen: :desc } 60 | t.index ["first_seen"], name: "index_ad_texts_on_first_seen" 61 | t.index ["last_seen"], name: "index_ad_texts_on_last_seen" 62 | t.index ["page_id"], name: "index_ad_texts_on_page_id" 63 | t.index ["paid_for_by"], name: "index_ad_texts_on_paid_for_by" 64 | t.index ["tsv"], name: "index_ad_texts_on_tsv", using: :gin 65 | end 66 | 67 | create_table "ad_topics", force: :cascade do |t| 68 | t.integer "topic_id" 69 | t.float "proportion" 70 | t.datetime "created_at", precision: 6, null: false 71 | t.datetime "updated_at", precision: 6, null: false 72 | t.integer "ad_text_id" 73 | end 74 | 75 | create_table "ads_local", id: false, force: :cascade do |t| 76 | t.text "ad_creative_body" 77 | t.datetime "ad_delivery_start_time", precision: 4 78 | t.datetime "ad_delivery_stop_time", precision: 4 79 | t.datetime "ad_creation_time", precision: 4 80 | t.bigint "page_id" 81 | t.string "currency", limit: 255 82 | t.string "ad_snapshot_url" 83 | t.boolean "is_active" 84 | t.integer "ad_sponsor_id" 85 | t.bigint "archive_id", null: false 86 | t.bigint "nyu_id", default: -> { "nextval('ads_nyu_id_seq'::regclass)" }, null: false 87 | t.string "ad_creative_link_caption" 88 | t.string "ad_creative_link_title" 89 | t.string "ad_creative_link_description" 90 | t.integer "ad_category_id" 91 | t.bigint "ad_id" 92 | t.string "country_code" 93 | t.boolean "most_recent" 94 | t.string "funding_entity" 95 | t.index ["archive_id"], name: "unique_ad_archive_id", unique: true 96 | end 97 | 98 | create_table "big_spenders", force: :cascade do |t| 99 | t.integer "ad_archive_report_id" 100 | t.integer "previous_ad_archive_report_id" 101 | t.integer "ad_archive_report_page_id" 102 | t.bigint "page_id" 103 | t.integer "spend_amount" 104 | t.integer "duration_days" 105 | t.boolean "is_new" 106 | t.datetime "created_at", precision: 6, null: false 107 | t.datetime "updated_at", precision: 6, null: false 108 | end 109 | 110 | create_table "demo_groups", id: :serial, force: :cascade do |t| 111 | t.string "age", limit: 255 112 | t.string "gender", limit: 255 113 | end 114 | 115 | create_table "demo_impressions_local", id: false, force: :cascade do |t| 116 | t.bigint "ad_archive_id" 117 | t.integer "demo_id" 118 | t.integer "min_impressions" 119 | t.integer "max_impressions" 120 | t.integer "min_spend" 121 | t.integer "max_spend" 122 | t.date "crawl_date" 123 | t.boolean "most_recent" 124 | t.bigint "nyu_id", default: -> { "nextval('demo_impressions_nyu_id1_seq'::regclass)" }, null: false 125 | t.index ["ad_archive_id", "demo_id"], name: "demo_impressions_unique_ad_archive_id", unique: true 126 | t.index ["ad_archive_id"], name: "demo_impressions_archive_id_idx" 127 | end 128 | 129 | create_table "fbpac_ads", id: :text, force: :cascade do |t| 130 | t.text "html", null: false 131 | t.integer "political", null: false 132 | t.integer "not_political", null: false 133 | t.text "title" 134 | t.text "message", null: false 135 | t.text "thumbnail", null: false 136 | t.datetime "created_at", default: -> { "now()" }, null: false 137 | t.datetime "updated_at", default: -> { "now()" }, null: false 138 | t.text "lang", null: false 139 | t.text "images", null: false, array: true 140 | t.integer "impressions", default: 1, null: false 141 | t.float "political_probability", default: 0.0, null: false 142 | t.text "targeting" 143 | t.boolean "suppressed", default: false, null: false 144 | t.jsonb "targets", default: [] 145 | t.text "advertiser" 146 | t.jsonb "entities", default: [] 147 | t.text "page" 148 | t.string "lower_page" 149 | t.text "targetings", array: true 150 | t.text "paid_for_by" 151 | t.integer "targetedness" 152 | t.decimal "listbuilding_fundraising_proba", precision: 9, scale: 6 153 | t.bigint "page_id" 154 | t.index ["advertiser"], name: "index_fbpac_ads_on_advertiser" 155 | t.index ["entities"], name: "index_fbpac_ads_on_entities", using: :gin 156 | t.index ["lang"], name: "index_fbpac_ads_on_browser_lang" 157 | t.index ["lower_page"], name: "fbpac_ads_lower_page_idx" 158 | t.index ["page"], name: "index_fbpac_ads_on_page" 159 | t.index ["political_probability"], name: "index_fbpac_ads_on_political_probability" 160 | t.index ["targets"], name: "index_fbpac_ads_on_targets", using: :gin 161 | end 162 | 163 | create_table "impressions_local", id: false, force: :cascade do |t| 164 | t.bigint "ad_archive_id" 165 | t.date "crawl_date" 166 | t.integer "min_impressions" 167 | t.integer "min_spend" 168 | t.integer "max_impressions" 169 | t.integer "max_spend" 170 | t.boolean "most_recent" 171 | t.bigint "nyu_id", default: -> { "nextval('impressions_nyu_id1_seq'::regclass)" }, null: false 172 | t.index ["ad_archive_id"], name: "impressions_archive_id_idx" 173 | t.index ["ad_archive_id"], name: "impressions_unique_ad_archive_id", unique: true 174 | end 175 | 176 | create_table "job_runs", force: :cascade do |t| 177 | t.integer "job_id" 178 | t.datetime "start_time" 179 | t.datetime "end_time" 180 | t.boolean "success" 181 | t.datetime "created_at", precision: 6, null: false 182 | t.datetime "updated_at", precision: 6, null: false 183 | end 184 | 185 | create_table "jobs", force: :cascade do |t| 186 | t.string "name" 187 | t.integer "expected_run_rate" 188 | t.decimal "estimated_duration" 189 | t.datetime "created_at", precision: 6, null: false 190 | t.datetime "updated_at", precision: 6, null: false 191 | end 192 | 193 | create_table "pages_local", id: false, force: :cascade do |t| 194 | t.string "page_name", limit: 255 195 | t.bigint "page_id" 196 | t.boolean "federal_candidate" 197 | t.string "url" 198 | t.boolean "is_deleted" 199 | end 200 | 201 | create_table "payers", force: :cascade do |t| 202 | t.string "name" 203 | t.text "notes" 204 | t.datetime "created_at", precision: 6, null: false 205 | t.datetime "updated_at", precision: 6, null: false 206 | end 207 | 208 | create_table "region_impressions_local", id: false, force: :cascade do |t| 209 | t.bigint "ad_archive_id" 210 | t.integer "region_id" 211 | t.integer "min_impressions" 212 | t.integer "min_spend" 213 | t.integer "max_impressions" 214 | t.integer "max_spend" 215 | t.date "crawl_date" 216 | t.boolean "most_recent" 217 | t.bigint "nyu_id", default: -> { "nextval('region_impressions_nyu_id1_seq'::regclass)" }, null: false 218 | t.index ["ad_archive_id", "region_id"], name: "region_impressions_unique_ad_archive_id", unique: true 219 | t.index ["ad_archive_id"], name: "region_impressions_archive_id_idx" 220 | end 221 | 222 | create_table "regions", id: :serial, force: :cascade do |t| 223 | t.string "name" 224 | end 225 | 226 | create_table "topics", force: :cascade do |t| 227 | t.string "topic" 228 | t.datetime "created_at", precision: 6, null: false 229 | t.datetime "updated_at", precision: 6, null: false 230 | end 231 | 232 | create_table "users", force: :cascade do |t| 233 | t.string "email", default: "", null: false 234 | t.string "encrypted_password", default: "", null: false 235 | t.string "reset_password_token" 236 | t.datetime "reset_password_sent_at" 237 | t.datetime "remember_created_at" 238 | t.datetime "created_at", precision: 6, null: false 239 | t.datetime "updated_at", precision: 6, null: false 240 | t.index ["email"], name: "index_users_on_email", unique: true 241 | t.index ["reset_password_token"], name: "index_users_on_reset_password_token", unique: true 242 | end 243 | 244 | create_table "writable_ads", force: :cascade do |t| 245 | t.string "partisanship" 246 | t.string "purpose" 247 | t.string "optimism" 248 | t.string "attack" 249 | t.bigint "archive_id" 250 | t.datetime "created_at", precision: 6, null: false 251 | t.datetime "updated_at", precision: 6, null: false 252 | t.string "text_hash" 253 | t.text "ad_id" 254 | t.bigint "page_id" 255 | t.boolean "swing_state_ad" 256 | t.text "states", default: [], array: true 257 | t.text "s3_url" 258 | t.index ["ad_id"], name: "index_writable_ads_on_ad_id" 259 | t.index ["archive_id"], name: "index_writable_ads_on_archive_id" 260 | t.index ["page_id"], name: "index_writable_ads_on_page_id" 261 | t.index ["text_hash"], name: "index_writable_ads_on_text_hash" 262 | end 263 | 264 | create_table "writable_pages", force: :cascade do |t| 265 | t.bigint "page_id" 266 | t.text "notes" 267 | t.datetime "created_at", precision: 6, null: false 268 | t.datetime "updated_at", precision: 6, null: false 269 | t.string "disclaimer" 270 | t.boolean "core", default: false 271 | t.string "partisanship" 272 | end 273 | 274 | end 275 | --------------------------------------------------------------------------------