├── .clang_complete
├── .gitignore
├── LICENCE
├── README.md
├── SConstruct
├── glossary
├── include
    └── rejit.h
├── sample
    ├── SConscript
    ├── basic.cc
    ├── jrep.cc
    ├── regexdna-multithread.cc
    └── regexdna.cc
├── src
    ├── allocation.h
    ├── assembler-base.cc
    ├── assembler-base.h
    ├── assembler.h
    ├── checks.cc
    ├── checks.h
    ├── codegen.cc
    ├── codegen.h
    ├── config.h
    ├── cpu.h
    ├── flags.cc
    ├── flags.h
    ├── globals.h
    ├── macro-assembler-base.h
    ├── macro-assembler.h
    ├── memory.h
    ├── parser.cc
    ├── parser.h
    ├── platform.h
    ├── platform
    │   ├── platform-linux.cc
    │   ├── platform-macos.cc
    │   ├── platform-posix.cc
    │   └── platform-posix.h
    ├── regexp.cc
    ├── regexp.h
    ├── rejit.cc
    ├── suffix_trees.cc
    ├── suffix_trees.h
    ├── utils.cc
    ├── utils.h
    └── x64
    │   ├── assembler-x64-inl.h
    │   ├── assembler-x64.cc
    │   ├── assembler-x64.h
    │   ├── codegen-x64.cc
    │   ├── macro-assembler-x64-inl.h
    │   ├── macro-assembler-x64.cc
    │   └── macro-assembler-x64.h
└── tools
    ├── analysis
        ├── SConscript
        └── compinfo.cc
    ├── benchmarks
        ├── SConscript
        ├── engines
        │   ├── bench_engine.cc
        │   ├── bench_engine.h
        │   ├── pcre
        │   │   ├── SConscript
        │   │   └── engine.cc
        │   ├── re2
        │   │   ├── SConscript
        │   │   └── engine.cc
        │   ├── rejit
        │   │   ├── SConscript
        │   │   └── engine.cc
        │   └── v8
        │   │   ├── SConscript
        │   │   └── engine
        ├── gjrep.py
        ├── resources
        │   ├── html
        │   │   ├── bench_plot.js
        │   │   ├── benchmarks_results.html.footer
        │   │   ├── benchmarks_results.html.header
        │   │   └── main.css
        │   └── sample_bench_complex.png
        └── run.py
    ├── debug
        └── lldb_rejit.py
    ├── tests
        ├── run.py
        └── test.cc
    └── utils.py


/.clang_complete:
--------------------------------------------------------------------------------
1 | -std=c++11 -Wall -pedantic -Werror -Isrc/ -Iinclude/ -DREJIT_TARGET_ARCH_X64 -g -DDEBUG
2 | 


--------------------------------------------------------------------------------
/.gitignore:
--------------------------------------------------------------------------------
 1 | *.o
 2 | *.pyc
 3 | *.swp
 4 | .DS_Store
 5 | .sconf_temp/
 6 | .sconsign.dblite
 7 | .tags
 8 | build
 9 | config.log
10 | tools/benchmarks/benchmarks_results.html
11 | tools/benchmarks/engines/pcre/engine
12 | tools/benchmarks/engines/pcre/svn.pcre/
13 | tools/benchmarks/engines/re2/engine
14 | tools/benchmarks/engines/re2/hg.re2/
15 | tools/benchmarks/engines/rejit/engine
16 | tools/benchmarks/resources/html/flot/
17 | 


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
  1 | Rejit is a prototype of a non-backtracking, just-in-time, SIMD-able regular
  2 | expression compiler developed on our free time. It is available under the GPLv3
  3 | licence. It currently only supports the x86_64 architecture.
  4 | 
  5 | ## Documentation
  6 | 
  7 | Documentation and information is available [here][coreperf rejit]. Below are
  8 | some sample benchmarks results.
  9 | 
 10 | ## Benchmarks
 11 | 
 12 | The results below were produced on a machine with a quad-core Intel Core
 13 | i5-2400 CPU @ 3.10GHz with 4GiB RAM on Fedora 17 (3.8.13-100.fc17.x86\_64).
 14 | It supports SSE4.2.
 15 | 
 16 | Results are reported for the following engines versions:
 17 | 
 18 | ```
 19 | GNU grep version 2.14, commit: 599f2b15bc152cdf19022c7803f9a5f336c25e65
 20 | Rejit commit: b29ea4af1a3ae86dcb25bf961bc716029430c9b1
 21 | V8 version 3.20.9, commit: 455bb9c2ab8af080aa15e0fbf4838731f45241e8
 22 | Re2 commit: aa957b5e3374
 23 | ```
 24 | 
 25 | #### Grepping recursively through the Linux kernel sources.
 26 | 
 27 | ```
 28 | $ CMD='grep -R regexp linux-3.10.6/'; $CMD > /dev/null && time $CMD > /dev/null
 29 | real  0m0.622s
 30 | user  0m0.356s
 31 | sys   0m0.260s
 32 | ```
 33 | 
 34 |  ```jrep``` is a grep-like utility powered by rejit.
 35 | 
 36 | ```
 37 | $ CMD='jrep -R regexp linux-3.10.6/'; $CMD > /dev/null && time $CMD > /dev/null
 38 | real  0m0.370s
 39 | user  0m0.101s
 40 | sys   0m0.263s
 41 | ```
 42 | 
 43 | The `jrep` utility performs 1.68 times faster than gnu grep in this very real
 44 | use-case!  The time spent in `sys` is equivalent, but Rejit spends 3 times less
 45 | time in `user` code.
 46 | <br />It is part of the sample programs in the rejit repository (see the
 47 | wiki).  It is of course far behind grep in terms of features, but
 48 | supports searching for multi-lines patterns and has initial multi-threading
 49 | support.
 50 | 
 51 | #### DNA matching benchmark.
 52 | 
 53 | From the "[Computer Language Benchmarks Game][2]", this benchmark performs some DNA matching operations using regular expressions.
 54 | 
 55 | The tables below show performance (`real` running time) for different input sizes, for
 56 | 
 57 | the
 58 | [fastest registered single threaded implementation][cpu_bench single threaded fastest]
 59 | (V8) and a single-threaded Rejit-powered implementation.
 60 | 
 61 | ```
 62 | input size                            V8      Rejit
 63 |     50.000 (500KB)                0.034s     0.015s
 64 |    500.000 (  5MB)                0.217s     0.130s
 65 |  5.000.000 ( 50MB)                2.054s     1.246s
 66 | 50.000.000 (500MB)       (out of memory)    14.624s
 67 | ```
 68 | 
 69 | the
 70 | [second fastest registered single threaded implementation][cpu_bench multi threaded fastest]
 71 | (Re2) and a multi-threaded Rejit-powered implementation. (A quick go at running
 72 | the first listed implementation would raise failures.)
 73 | 
 74 | ```
 75 | input size                           Re2      Rejit
 76 |     50.000 (500KB)                0.022s     0.011s
 77 |    500.000 (  5MB)                0.183s     0.087s
 78 |  5.000.000 ( 50MB)                1.629s     0.971s
 79 | 50.000.000 (500MB)               20.693s    11.594s
 80 | ```
 81 | 
 82 | See performance for various engines and languages for [single-core][4] and [quad-core][5] implementations.
 83 | The rejit programs used to run these benchmarks are also part of the rejit
 84 | sample programs (see the wiki).
 85 | 
 86 | #### Complex regular expression matching
 87 | This is an example taken from rejit's benchmarks suite. It shows the performance to find all left-most longest matches of the regular expression ```([complex]|(regexp)){2,7}abcdefgh(at|the|[e-nd]as well)``` in randomly generated texts of various sizes. The performance reported is ```(<size of text> / <time to match>)```, which is easier to report and understand than close to zero time intervals.
 88 | 
 89 | It illustrates the 'fast forward' mechanism used by rejit (see [this article][coreperf rejit] for details).<br />
 90 | 
 91 | ![Graph comparing performance of re2, v8, and rejit to match a complex regular expression][6]
 92 | 
 93 | 
 94 | 
 95 |   [2]: http://benchmarksgame.alioth.debian.org/
 96 |   [3]: http://benchmarksgame.alioth.debian.org/u64/program.php?test=regexdna&lang=v8&id=2
 97 |   [4]: http://benchmarksgame.alioth.debian.org/u64/benchmark.php?test=regexdna&lang=all&data=u64
 98 |   [5]: http://benchmarksgame.alioth.debian.org/u64q/benchmark.php?test=regexdna&lang=all&data=u64q
 99 |   [6]: tools/benchmarks/resources/sample_bench_complex.png
100 |   [cpu_bench single threaded fastest]: http://benchmarksgame.alioth.debian.org/u64/program.php?test=regexdna&lang=v8&id=2
101 |   [cpu_bench multi threaded fastest]: http://benchmarksgame.alioth.debian.org/u32q/program.php?test=regexdna&lang=gpp&id=2
102 |   [coreperf rejit]: http://coreperf.com/projects/rejit/
103 | 


--------------------------------------------------------------------------------
/glossary:
--------------------------------------------------------------------------------
 1 | mc:
 2 |   MultipleChar.
 3 |   Instance of the MultipleChar class, which represents a sequence of non special
 4 |   characters in the regular expression.
 5 | 
 6 | sol / eol:
 7 |   start / end of line.
 8 |   Instance of the Start/EndOfLine class, which represent '^' or '$' in a regexp.
 9 | 
10 | sos / eos:
11 |   start / end of string.
12 |   Pointer to the start or terminating character '\0' of a string.
13 | 
14 | seos / seol: short for sos/eos / sol/eol
15 | 


--------------------------------------------------------------------------------
/include/rejit.h:
--------------------------------------------------------------------------------
  1 | // Copyright (C) 2013 Alexandre Rames <alexandre@coreperf.com>
  2 | // rejit is free software: you can redistribute it and/or modify
  3 | // it under the terms of the GNU General Public License as published by
  4 | // the Free Software Foundation, either version 3 of the License, or
  5 | // (at your option) any later version.
  6 | //
  7 | // This program is distributed in the hope that it will be useful,
  8 | // but WITHOUT ANY WARRANTY; without even the implied warranty of
  9 | // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 10 | // GNU General Public License for more details.
 11 | //
 12 | // You should have received a copy of the GNU General Public License
 13 | // along with this program.  If not, see <http://www.gnu.org/licenses/>.
 14 | 
 15 | // rejit headers.
 16 | 
 17 | #ifndef REJIT_H_
 18 | #define REJIT_H_
 19 | 
 20 | #include <string>
 21 | #include <vector>
 22 | 
 23 | using namespace std;
 24 | 
 25 | namespace rejit {
 26 | 
 27 | // A match result.
 28 | // Upon match,
 29 | // begin points to the first character of the match.
 30 | // end points to
 31 | //  - the first character of the match if the match has a length of 0.
 32 | //  - one character past the last character of the match otherwise.
 33 | //
 34 | // For example:
 35 | //  - When matching '^$' in an empty string "", the first (and only) match
 36 | //  will have both begin and end pointing at the end of string marker.
 37 | //  - When matching 'abc' in string "0abc1", the first (and only) match
 38 | //  will have begin pointing at 'a' and end pointing at '1'.
 39 | //
 40 | // TODO: Consider using offsets rather than pointers in the Match structure.
 41 | struct Match {
 42 |   const char* begin;
 43 |   const char* end;
 44 | };
 45 | 
 46 | // High level helpers.
 47 | // These are convenient helpers that abstract the use of the Regej class below.
 48 | // Rejit currently does not yet have a cache for compiled regular expressions,
 49 | // so using the Regej class methods will be more efficient than calling these
 50 | // multiple times.
 51 | // TODO(rames): Add table with examples.
 52 | 
 53 | // Returns true iff the regexp matches the whole text.
 54 | bool MatchFull(const char* regexp, const string& text);
 55 | bool MatchFull(const char* regexp, const char* text, size_t text_size);
 56 | // Returns true if there is a match in the text.
 57 | bool MatchAnywhere(const char* regexp, const string& text);
 58 | bool MatchAnywhere(const char* regexp, const char* text, size_t text_size);
 59 | // Find the left-most longest match in the text. Returns true if there is a
 60 | // match, and false otherwise.
 61 | bool MatchFirst(const char* regexp, const string& text, Match* match);
 62 | bool MatchFirst(const char* regexp, const char* text, size_t text_size, Match* match);
 63 | // Fill the vector with all left-most longest matches.
 64 | // Returns the size of the vector for convenience.
 65 | size_t MatchAll(const char* regexp, const string& text,
 66 |                 std::vector<struct Match>* matches);
 67 | size_t MatchAll(const char* regexp, const char* text, size_t text_size,
 68 |                 std::vector<struct Match>* matches);
 69 | // Count the number of left-most longest matches in the text.
 70 | size_t MatchAllCount(const char* regexp, const string& text);
 71 | size_t MatchAllCount(const char* regexp, const char* text, size_t text_size);
 72 | 
 73 | // Replace one or multiple matches in text.
 74 | void Replace(Match to_replace, string& text, const string& with);
 75 | void Replace(vector<Match>* to_replace, string& text, const string& with);
 76 | // This is equivalent to MatchFirst/MatchAll followed by Replace.
 77 | // Returns the number of matches that were replaced.
 78 | bool ReplaceFirst(const char* regexp, string& text, const string& with);
 79 | size_t ReplaceAll(const char* regexp, string& text, const string& with);
 80 | 
 81 | // Types of matches. 
 82 | // Ordered by matching 'difficulty'.
 83 | enum MatchType {
 84 |   kMatchFull,
 85 |   kMatchAnywhere,
 86 |   kMatchFirst,
 87 |   kMatchAll,
 88 |   kNMatchTypes
 89 | };
 90 | namespace internal  {
 91 | // Internal structure used to track compilation information.
 92 | // A forward declaration is required here to reference it from class Regej.
 93 | class RegexpInfo;
 94 | }
 95 | 
 96 | // Error status returned by rejit.
 97 | // Upon error, the rejit_status_string is updated with an error message.
 98 | enum Status {
 99 |   RejitSuccess = 0,
100 |   // Parser errors must have negative codes.
101 |   ParserError = -1
102 | };
103 | extern char* const rejit_status_string;
104 | 
105 | class Regej {
106 |  public:
107 |   explicit Regej(const char* regexp);
108 |   explicit Regej(const string& regexp);
109 |   ~Regej();
110 | 
111 |   // Error codes used to indicate the status of the Regej.
112 |   Status status() const { return status_; }
113 | 
114 |   // See related global functions above for details about the following
115 |   // functions.
116 |   bool MatchFull(const string& text);
117 |   bool MatchFull(const char* text, size_t text_size);
118 |   bool MatchAnywhere(const string& text);
119 |   bool MatchAnywhere(const char* text, size_t text_size);
120 |   bool MatchFirst(const string& text, Match* match);
121 |   bool MatchFirst(const char* text, size_t text_size, Match* match);
122 |   size_t MatchAll(const string& text, std::vector<struct Match>* matches);
123 |   size_t MatchAll(const char* text, size_t text_size, std::vector<struct Match>* matches);
124 |   size_t MatchAllCount(const string& text);
125 |   size_t MatchAllCount(const char* text, size_t text_size);
126 | 
127 |   // This is equivalent to MatchFirst/MatchAll followed by Replace.
128 |   bool ReplaceFirst(string& text, const string& with);
129 |   size_t ReplaceAll(string& text, const string& with);
130 | 
131 |   bool Compile(MatchType match_type);
132 | 
133 |  private:
134 |   char const * const regexp_;
135 |   // This refers to internal compilation information.
136 |   internal::RegexpInfo* rinfo_;
137 |   Status status_;
138 | };
139 | 
140 | }  // namespace rejit
141 | 
142 | #endif  // REJIT_H_
143 | 


--------------------------------------------------------------------------------
/sample/SConscript:
--------------------------------------------------------------------------------
 1 | # Copyright (C) 2013 Alexandre Rames <alexandre@coreperf.com>
 2 | 
 3 | # rejit is free software: you can redistribute it and/or modify
 4 | # it under the terms of the GNU General Public License as published by
 5 | # the Free Software Foundation, either version 3 of the License, or
 6 | # (at your option) any later version.
 7 | #
 8 | # This program is distributed in the hope that it will be useful,
 9 | # but WITHOUT ANY WARRANTY; without even the implied warranty of
10 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
11 | # GNU General Public License for more details.
12 | #
13 | # You should have received a copy of the GNU General Public License
14 | # along with this program.  If not, see <http://www.gnu.org/licenses/>.
15 | 
16 | import os
17 | from os.path import join, dirname
18 | 
19 | Import('env librejit')
20 | 
21 | 
22 | basic = env.Program('basic', 'basic.cc', LIBS=[librejit])
23 | env.Alias('basic', basic)
24 | regexdna = env.Program('regexdna', 'regexdna.cc', LIBS=[librejit])
25 | env.Alias('regexdna', regexdna)
26 | regexdna_multithread_LINKFLAGS = list(env['LINKFLAGS'])
27 | if env['os'] == 'linux':
28 |   regexdna_multithread_LINKFLAGS += ['-pthread']
29 | regexdna_multithread = env.Program('regexdna-multithread', 'regexdna-multithread.cc', LIBS=[librejit], LINKFLAGS=regexdna_multithread_LINKFLAGS)
30 | env.Alias('regexdna-multithread', regexdna_multithread)
31 | 
32 | jrep_libs = [librejit]
33 | if ('sample/jrep' in COMMAND_LINE_TARGETS or 'jrep' in COMMAND_LINE_TARGETS) and 'os' in env and env['os'] == 'macos':
34 |   conf = Configure(env)
35 |   if not conf.CheckLib('argp'):
36 |     print 'help argp'
37 |     Exit(1)
38 |   env = conf.Finish()
39 |   jrep_libs += ['argp']
40 | jrep_CXXFLAGS = list(env['CXXFLAGS'])
41 | jrep_LINKFLAGS = list(env['LINKFLAGS'])
42 | if env['os'] == 'linux':
43 |   jrep_CXXFLAGS += ['-D_GLIBCXX_GTHREAD_USE_WEAK=0']
44 |   jrep_LINKFLAGS += ['-pthread']
45 | jrep = env.Program('jrep', 'jrep.cc', LIBS=jrep_libs, CXXFLAGS=jrep_CXXFLAGS, LINKFLAGS=jrep_LINKFLAGS)
46 | env.Alias('jrep', jrep)
47 | 


--------------------------------------------------------------------------------
/sample/basic.cc:
--------------------------------------------------------------------------------
 1 | // Copyright (C) 2013 Alexandre Rames <alexandre@coreperf.com>
 2 | // rejit is free software: you can redistribute it and/or modify
 3 | // it under the terms of the GNU General Public License as published by
 4 | // the Free Software Foundation, either version 3 of the License, or
 5 | // (at your option) any later version.
 6 | //
 7 | // This program is distributed in the hope that it will be useful,
 8 | // but WITHOUT ANY WARRANTY; without even the implied warranty of
 9 | // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
10 | // GNU General Public License for more details.
11 | //
12 | // You should have received a copy of the GNU General Public License
13 | // along with this program.  If not, see <http://www.gnu.org/licenses/>.
14 | 
15 | #include <stdio.h>
16 | #include <stdlib.h>
17 | // For file operations.
18 | #include <fcntl.h>
19 | #include <sys/mman.h>
20 | #include <sys/stat.h>
21 | 
22 | #include "rejit.h"
23 | 
24 | using namespace std;
25 | 
26 | int main() {
27 |   printf("Running basic rejit example.\n");
28 | 
29 |   const char* regexp = "(right|[ts]ion)";
30 |   const char* text;
31 |   int fd;
32 |   // Get the text in the file.
33 |   struct stat file_stats;
34 |   fd = open("LICENCE", O_RDONLY);
35 |   fstat(fd, &file_stats);
36 |   text = (char*)mmap(NULL, file_stats.st_size,
37 |                      PROT_READ, MAP_PRIVATE, fd, 0);
38 | 
39 |   printf("Searching for %s in the LICENCE file.\n", regexp);
40 | 
41 |   rejit::Regej re(regexp);
42 |   vector<rejit::Match> matches;
43 | 
44 |   re.MatchAll(text, &matches);
45 | 
46 |   printf("Found %ld matches.\n", matches.size());
47 | 
48 |   printf("Printing the first 10:\n");
49 |   unsigned i = 0;
50 |   vector<rejit::Match>::iterator it;
51 |   for (it = matches.begin(); it < matches.end() && i++ < 10; it++) {
52 |     const char* c = (*it).begin;
53 |     while (c < (*it).end) {
54 |       printf("%c", *c);
55 |       c++;
56 |     }
57 |     printf("\n");
58 |   }
59 | 
60 |   munmap((void*)text, file_stats.st_size);
61 | 
62 |   return EXIT_SUCCESS;
63 | }
64 | 


--------------------------------------------------------------------------------
/sample/regexdna-multithread.cc:
--------------------------------------------------------------------------------
  1 | // Copyright (C) 2013 Alexandre Rames <alexandre@coreperf.com>
  2 | // rejit is free software: you can redistribute it and/or modify
  3 | // it under the terms of the GNU General Public License as published by
  4 | // the Free Software Foundation, either version 3 of the License, or
  5 | // (at your option) any later version.
  6 | //
  7 | // This program is distributed in the hope that it will be useful,
  8 | // but WITHOUT ANY WARRANTY; without even the implied warranty of
  9 | // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 10 | // GNU General Public License for more details.
 11 | //
 12 | // You should have received a copy of the GNU General Public License
 13 | // along with this program.  If not, see <http://www.gnu.org/licenses/>.
 14 | 
 15 | 
 16 | #include "rejit.h"
 17 | 
 18 | #include <stdio.h>
 19 | #include <stdlib.h>
 20 | #include <string>
 21 | #include <thread>
 22 | #include <atomic>
 23 | #include <algorithm>
 24 | 
 25 | using namespace std;
 26 | 
 27 | char const * const dna8mers[] = {
 28 |   "agggtaaa|tttaccct",
 29 |   "[cgt]gggtaaa|tttaccc[acg]",
 30 |   "a[act]ggtaaa|tttacc[agt]t",
 31 |   "ag[act]gtaaa|tttac[agt]ct",
 32 |   "agg[act]taaa|ttta[agt]cct",
 33 |   "aggg[acg]aaa|ttt[cgt]ccct",
 34 |   "agggt[cgt]aa|tt[acg]accct",
 35 |   "agggta[cgt]a|t[acg]taccct",
 36 |   "agggtaa[cgt]|[acg]ttaccct"
 37 | };
 38 | 
 39 | char const * const iub_codes[] = {
 40 |   "B", "(c|g|t)",
 41 |   "D", "(a|g|t)",
 42 |   "H", "(a|c|t)",
 43 |   "K", "(g|t)",
 44 |   "M", "(a|c)",
 45 |   "N", "(a|c|g|t)",
 46 |   "R", "(a|g)",
 47 |   "S", "(c|g)",
 48 |   "V", "(a|c|g)",
 49 |   "W", "(a|t)",
 50 |   "Y", "(c|t)"
 51 | };
 52 | 
 53 | // Use global variables to simplify the implementation of the thread functions.
 54 | const unsigned n_mers = sizeof(dna8mers) / sizeof(dna8mers[0]);
 55 | const unsigned n_iub_codes = sizeof(iub_codes) / sizeof(iub_codes[0]) / 2;
 56 | unsigned counts[n_mers];
 57 | vector<rejit::Match> iub_codes_matches[n_iub_codes];
 58 | string text;
 59 | 
 60 | thread *threads;
 61 | atomic_uint processed_mers = ATOMIC_VAR_INIT(0);
 62 | atomic_uint processed_iub_codes = ATOMIC_VAR_INIT(0);
 63 | 
 64 | 
 65 | void thread_count_mers() {
 66 |   unsigned index;
 67 |   while ((index = processed_mers++) < n_mers) {
 68 |     counts[index] = rejit::MatchAllCount(dna8mers[index], text);
 69 |   }
 70 | }
 71 | 
 72 | 
 73 | void thread_find_iub_codes() {
 74 |   unsigned index;
 75 |   while ((index = processed_iub_codes++) < n_iub_codes) {
 76 |     rejit::MatchAll(iub_codes[2 * index], text, &iub_codes_matches[index]);
 77 |   }
 78 | }
 79 | 
 80 | 
 81 | bool cmp_match_early(rejit::Match a, rejit::Match b) {
 82 |   return a.begin < b.begin;
 83 | }
 84 | 
 85 | 
 86 | int main(int argc, char **argv) {
 87 |   char* text_raw;
 88 |   size_t text_raw_size, text_size, replaced_text_size;
 89 | 
 90 |   if (argc > 1) {
 91 |     printf(
 92 |         "Usage:\n"
 93 |         "  regexdna < input.file\n"
 94 |         "You can generate input files using the fasta program available from\n"
 95 |         "  http://benchmarksgame.alioth.debian.org/u64/benchmark.php?test=fasta&lang=gcc&id=1&data=u64"
 96 |         );
 97 |     return EXIT_SUCCESS;
 98 |   }
 99 | 
100 |   // Find the number of threads to use.
101 |   unsigned n_threads = thread::hardware_concurrency();
102 |   if (n_threads == 0) n_threads = 4;
103 |   threads = reinterpret_cast<thread*>(malloc(n_threads * sizeof(thread)));
104 | 
105 |   // Initialize the text.
106 |   fseek(stdin, 0, SEEK_END);
107 |   text_raw_size = ftell(stdin);
108 |   text_raw = (char*)malloc(text_raw_size + 1);
109 |   rewind(stdin);
110 |   fread(text_raw, 1, text_raw_size, stdin);
111 |   text.assign(text_raw, text_raw_size);
112 |   free(text_raw);
113 | 
114 |   rejit::ReplaceAll(">.*\n|\n", text, "");
115 |   text_size = text.size();
116 | 
117 |   // Count all mers.
118 |   for (unsigned i = 0; i < n_threads; i++) {
119 |     threads[i] = thread(thread_count_mers);
120 |   }
121 |   for (unsigned i = 0; i < n_threads; i++) {
122 |     threads[i].join();
123 |   }
124 |   for (unsigned i = 0; i < n_mers; i++) {
125 |     printf("%s %d\n", dna8mers[i], counts[i]);
126 |   }
127 | 
128 |   // Search for all iub_codes to replace.
129 |   for (unsigned i = 0; i < n_threads; i++) {
130 |     threads[i] = thread(thread_find_iub_codes);
131 |   }
132 |   for (unsigned i = 0; i < n_threads; i++) {
133 |     threads[i].join();
134 |   }
135 |   // Merge all iub_codes matches into one vector and sort it.
136 |   vector<rejit::Match> *total_matches = &iub_codes_matches[0];
137 |   size_t total_matches_size = 0;
138 |   for (unsigned i = 0; i < n_iub_codes; i ++) {
139 |     total_matches_size += iub_codes_matches[i].size();
140 |   }
141 |   total_matches->reserve(total_matches_size);
142 |   for (unsigned i = 1; i < n_iub_codes; i ++) {
143 |     total_matches->insert(total_matches->end(),
144 |                           iub_codes_matches[i].begin(), iub_codes_matches[i].end());
145 |   }
146 |   sort(total_matches->begin(), total_matches->end(), cmp_match_early);
147 | 
148 |   // Replace all matches, looking for the correct replacement in the array.
149 |   vector<rejit::Match>::iterator it;
150 |   string res;
151 |   // We reserve 6 extra characters per replaced match.
152 |   // The replacing strings are on average less than 7 characters, minus one
153 |   // character that is being replaced.
154 |   res.reserve(text.size() + 6 * total_matches->size());
155 |   const char* ptext = text.c_str();
156 |   for (it = total_matches->begin(); it < total_matches->end(); it++) {
157 |     res.append(ptext, (*it).begin - ptext);
158 |     ptext = (*it).end;
159 |     // Replace with the correct pattern.
160 |     for (unsigned i = 0; i < n_iub_codes; i++) {
161 |       if (*(it->begin) == iub_codes[2 * i][0]) {
162 |         res.append(iub_codes[2 * i + 1]);
163 |         break;
164 |       }
165 |     }
166 |   }
167 |   res.append(ptext, text.c_str() + text.size() - ptext);
168 | 
169 |   replaced_text_size = res.size();
170 | 
171 |   printf("\n%u\n%u\n%u\n",
172 |          (unsigned)text_raw_size,
173 |          (unsigned)text_size,
174 |          (unsigned)replaced_text_size);
175 | 
176 |   free(threads);
177 | 
178 |   return EXIT_SUCCESS;
179 | }
180 | 


--------------------------------------------------------------------------------
/sample/regexdna.cc:
--------------------------------------------------------------------------------
 1 | // Copyright (C) 2013 Alexandre Rames <alexandre@coreperf.com>
 2 | // rejit is free software: you can redistribute it and/or modify
 3 | // it under the terms of the GNU General Public License as published by
 4 | // the Free Software Foundation, either version 3 of the License, or
 5 | // (at your option) any later version.
 6 | //
 7 | // This program is distributed in the hope that it will be useful,
 8 | // but WITHOUT ANY WARRANTY; without even the implied warranty of
 9 | // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
10 | // GNU General Public License for more details.
11 | //
12 | // You should have received a copy of the GNU General Public License
13 | // along with this program.  If not, see <http://www.gnu.org/licenses/>.
14 | 
15 | 
16 | #include "rejit.h"
17 | 
18 | #include <stdio.h>
19 | #include <stdlib.h>
20 | #include <string>
21 | #include <iostream>
22 | 
23 | using namespace std;
24 | 
25 | int main(int argc, char **argv) {
26 | 
27 |   char* text_raw;
28 |   string text;
29 |   size_t text_raw_size, text_size, replaced_text_size;
30 | 
31 |   if (argc > 1) {
32 |     printf(
33 |         "Usage:\n"
34 |         "  regexdna < input.file\n"
35 |         "You can generate input files using the fasta program available from\n"
36 |         "  http://benchmarksgame.alioth.debian.org/u64/benchmark.php?test=fasta&lang=gcc&id=1&data=u64"
37 |         );
38 |     return EXIT_SUCCESS;
39 |   }
40 | 
41 |   fseek(stdin, 0, SEEK_END);
42 |   text_raw_size = ftell(stdin);
43 |   text_raw = (char*)malloc(text_raw_size + 1);
44 |   rewind(stdin);
45 |   fread(text_raw, 1, text_raw_size, stdin);
46 |   text.assign(text_raw, text_raw_size);
47 |   free(text_raw);
48 | 
49 |   rejit::ReplaceAll(">.*\n|\n", text, "");
50 |   text_size = text.size();
51 | 
52 |   const char* dna8mers[] = {
53 |     "agggtaaa|tttaccct",
54 |     "[cgt]gggtaaa|tttaccc[acg]",
55 |     "a[act]ggtaaa|tttacc[agt]t",
56 |     "ag[act]gtaaa|tttac[agt]ct",
57 |     "agg[act]taaa|ttta[agt]cct",
58 |     "aggg[acg]aaa|ttt[cgt]ccct",
59 |     "agggt[cgt]aa|tt[acg]accct",
60 |     "agggta[cgt]a|t[acg]taccct",
61 |     "agggtaa[cgt]|[acg]ttaccct"
62 |   };
63 | 
64 |   for (unsigned i = 0; i < sizeof(dna8mers) / sizeof(char*); i++) {
65 |     unsigned count = rejit::MatchAllCount(dna8mers[i], text);
66 |     printf("%s %d\n", dna8mers[i], count);
67 |   }
68 | 
69 |   const char* iub_codes[] = {
70 |     "B", "(c|g|t)",
71 |     "D", "(a|g|t)",
72 |     "H", "(a|c|t)",
73 |     "K", "(g|t)",
74 |     "M", "(a|c)",
75 |     "N", "(a|c|g|t)",
76 |     "R", "(a|g)",
77 |     "S", "(c|g)",
78 |     "V", "(a|c|g)",
79 |     "W", "(a|t)",
80 |     "Y", "(c|t)"
81 |   };
82 | 
83 |   for (unsigned i = 0; i < sizeof(iub_codes) / sizeof(char*); i += 2) {
84 |     rejit::ReplaceAll(iub_codes[i], text, iub_codes[i + 1]);
85 |   }
86 |   replaced_text_size = text.size();
87 | 
88 |   printf("\n%u\n%u\n%u\n",
89 |          (unsigned)text_raw_size,
90 |          (unsigned)text_size,
91 |          (unsigned)replaced_text_size);
92 | 
93 |   return EXIT_SUCCESS;
94 | }
95 | 


--------------------------------------------------------------------------------
/src/allocation.h:
--------------------------------------------------------------------------------
 1 | // Copyright 2012 the V8 project authors. All rights reserved.
 2 | // Redistribution and use in source and binary forms, with or without
 3 | // modification, are permitted provided that the following conditions are
 4 | // met:
 5 | //
 6 | //     * Redistributions of source code must retain the above copyright
 7 | //       notice, this list of conditions and the following disclaimer.
 8 | //     * Redistributions in binary form must reproduce the above
 9 | //       copyright notice, this list of conditions and the following
10 | //       disclaimer in the documentation and/or other materials provided
11 | //       with the distribution.
12 | //     * Neither the name of Google Inc. nor the names of its
13 | //       contributors may be used to endorse or promote products derived
14 | //       from this software without specific prior written permission.
15 | //
16 | // THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
17 | // "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
18 | // LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
19 | // A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
20 | // OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
21 | // SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
22 | // LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
23 | // DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
24 | // THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
25 | // (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
26 | // OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
27 | 
28 | #ifndef REJIT_ALLOCATION_H_
29 | #define REJIT_ALLOCATION_H_
30 | 
31 | namespace rejit {
32 | namespace internal {
33 | 
34 | template <typename T>
35 | T* NewArray(size_t size) {
36 |   T* result = new T[size];
37 |   if (result == NULL) FATAL("Out of memory");
38 |   return result;
39 | }
40 | 
41 | 
42 | } }  // namespace rejit::internal
43 | 
44 | #endif  // REJIT_ALLOCATION_H_
45 | 


--------------------------------------------------------------------------------
/src/assembler-base.cc:
--------------------------------------------------------------------------------
  1 | // Copyright (c) 1994-2006 Sun Microsystems Inc.
  2 | // All Rights Reserved.
  3 | //
  4 | // Redistribution and use in source and binary forms, with or without
  5 | // modification, are permitted provided that the following conditions are
  6 | // met:
  7 | //
  8 | // - Redistributions of source code must retain the above copyright notice,
  9 | // this list of conditions and the following disclaimer.
 10 | //
 11 | // - Redistribution in binary form must reproduce the above copyright
 12 | // notice, this list of conditions and the following disclaimer in the
 13 | // documentation and/or other materials provided with the distribution.
 14 | //
 15 | // - Neither the name of Sun Microsystems or the names of contributors may
 16 | // be used to endorse or promote products derived from this software without
 17 | // specific prior written permission.
 18 | //
 19 | // THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
 20 | // IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
 21 | // THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
 22 | // PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
 23 | // CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
 24 | // EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
 25 | // PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
 26 | // PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
 27 | // LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
 28 | // NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
 29 | // SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 30 | 
 31 | // The original source code covered by the above license above has been
 32 | // modified significantly by Google Inc.
 33 | // Copyright 2012 the V8 project authors. All rights reserved.
 34 | 
 35 | #include "assembler-base.h"
 36 | 
 37 | #include "allocation.h"
 38 | 
 39 | #ifdef REJIT_TARGET_ARCH_X64
 40 | #else
 41 | #error "Unsupported architecture."
 42 | #endif
 43 | 
 44 | namespace rejit {
 45 | namespace internal {
 46 | 
 47 | 
 48 | // -----------------------------------------------------------------------------
 49 | // Implementation of Label
 50 | 
 51 | int Label::pos() const {
 52 |   if (pos_ < 0) return -pos_ - 1;
 53 |   if (pos_ > 0) return  pos_ - 1;
 54 |   UNREACHABLE();
 55 |   return 0;
 56 | }
 57 | 
 58 | 
 59 | RelocatedData::RelocatedData(char *buf, int buf_size, bool copy_buf,
 60 |                              int alignment) {
 61 |   alignment_ = alignment;
 62 |   if (!copy_buf) {
 63 |     buffer_size_ = buf_size;
 64 |     buffer_ = buf;
 65 |     own_buffer_ = false;
 66 |   } else {
 67 |     buffer_ = NewArray<char>(buf_size);
 68 |     buffer_size_ = buf_size;
 69 |     own_buffer_ = true;
 70 |     memcpy(buffer_, buf, buf_size);
 71 |   }
 72 | }
 73 | 
 74 | 
 75 | RelocatedData::~RelocatedData() {
 76 |   if (own_buffer_) {
 77 |     delete buffer_;
 78 |   }
 79 | }
 80 | 
 81 | 
 82 | AssemblerBase::AssemblerBase(size_t min_buffer_size, size_t max_buffer_size,
 83 |                              unsigned max_instr_size,
 84 |                              void* buffer, int buffer_size) :
 85 |     min_buffer_size_(min_buffer_size), max_buffer_size_(max_buffer_size),
 86 |     max_instr_size_(max_instr_size)
 87 | {
 88 |   if (buffer == NULL) {
 89 |     // Do our own buffer management.
 90 |     if (buffer_size <= min_buffer_size_) {
 91 |       buffer_size = min_buffer_size_;
 92 |     }
 93 |     buffer_ = NewArray<byte>(buffer_size);
 94 |     buffer_size_ = buffer_size;
 95 |     own_buffer_ = true;
 96 |   } else {
 97 |     // Use externally provided buffer instead.
 98 |     ASSERT(buffer_size > 0);
 99 |     buffer_ = static_cast<byte*>(buffer);
100 |     buffer_size_ = buffer_size;
101 |     own_buffer_ = false;
102 |   }
103 | 
104 |   // Set up buffer pointers.
105 |   ASSERT(buffer_ != NULL);
106 |   pc_ = buffer_;
107 | }
108 | 
109 | 
110 | AssemblerBase::~AssemblerBase() {
111 |   if (own_buffer_) {
112 |     delete[] buffer_;
113 |   }
114 |   for (RelocatedData* reloc : reloc_data_owned_) {
115 |     delete reloc;
116 |   }
117 | }
118 | 
119 | 
120 | // TODO(ajr): Clean and check memory type (writable?).
121 | VirtualMemory* AssemblerBase::GetCode() {
122 |   VirtualMemory* vmem = new VirtualMemory(pc_offset());
123 |   if (!vmem->IsReserved()) {
124 |     FATAL("VirtualMemory has not been reserved.");
125 |     delete vmem;
126 |     return NULL;
127 |   }
128 |   if (!vmem->Commit(vmem->address(), pc_offset(), true/*executable*/)) {
129 |     FATAL("VirtualMemory has not been committed.");
130 |     delete vmem;
131 |     return NULL;
132 |   }
133 | 
134 |   memcpy(vmem->address(), buffer_, pc_offset());
135 | 
136 |   // TODO(rames): Clean up how the code is returned. Returning virtual memory
137 |   // is not very intuitive!
138 |   return vmem;
139 | }
140 | 
141 | 
142 | void AssemblerBase::GrowBuffer(bool force) {
143 |   ASSERT(force || buffer_overflow());
144 |   if (!own_buffer_) {
145 |     FATAL("external code buffer is too small");
146 |   }
147 | 
148 |   byte* new_buffer;
149 |   size_t new_buffer_size;
150 | 
151 |   ASSERT(buffer_size_ >= min_buffer_size_);
152 |   if (buffer_size_ >= max_buffer_size_) {
153 |     FATAL("The code generation buffer has exceeded its maximum possible size.");
154 |   }
155 |   new_buffer_size = min(2 * buffer_size_, max_buffer_size_);
156 | 
157 |   // Set up new buffer.
158 |   new_buffer = NewArray<byte>(new_buffer_size);
159 | 
160 |   // Get the pc offset before switching buffers.
161 |   ptrdiff_t offset = pc_offset();
162 | 
163 |   // Copy the data.
164 |   memmove(new_buffer, buffer_, offset);
165 |   // Switch buffers.
166 |   delete[] buffer_;
167 |   buffer_ = new_buffer;
168 |   buffer_size_ = new_buffer_size;
169 |   pc_ = new_buffer + offset;
170 | 
171 |   ASSERT(!buffer_overflow());
172 | }
173 | 
174 | 
175 | RelocatedData *AssemblerBase::NewRelocatedData(char *buf, size_t buf_size,
176 |                                                bool copy_buf, int alignment) {
177 |   RelocatedData *reloc = new RelocatedData(buf, buf_size,
178 |                                            copy_buf, alignment);
179 |   reloc_data_owned_.push_back(reloc);
180 |   return reloc;
181 | }
182 | 
183 | 
184 | void AssemblerBase::EmitRelocData() {
185 |   // Emit relocation data that has not been emitted yet.
186 |   //for (pair<RelocatedData*, int>& reloc_info : reloc_data_location_) {
187 |   for (auto& reloc_info : reloc_data_location_) {
188 |     if (reloc_info.second == -1) {
189 |       RelocatedData* reloc = reloc_info.first;
190 |       if (available_space() < reloc->buffer_size_ + reloc->alignment_) {
191 |         GrowBuffer(true);
192 |       }
193 |       ASSERT(IsPowerOf2(reloc->alignment_));
194 |       int alignment_mask = reloc->alignment_ - 1;
195 |       pc_ += (reloc->alignment_ - (pc_offset() & alignment_mask)) & alignment_mask;
196 |       memcpy(pc_, reloc->buffer_, reloc->buffer_size_);
197 |       reloc_info.second = pc_offset();
198 |       pc_ += reloc->buffer_size_;
199 |     }
200 |   }
201 | }
202 | 
203 | 
204 | void AssemblerBase::UseRelocatedData(RelocatedData *data) {
205 |   map<RelocatedData*, int>::iterator it;
206 |   it = reloc_data_location_.find(data);
207 |   if (it == reloc_data_location_.end()) {
208 |     reloc_data_location_.insert(pair<RelocatedData*, int>(data, -1));
209 |   }
210 | }
211 | 
212 | 
213 | void AssemblerBase::UseRelocatedValue(RelocatedValue reloc) {
214 |   UseRelocatedData(reloc.data_);
215 |   reloc_values_usage_location_.insert(pair<RelocatedValue, int>(reloc, pc_offset()));
216 | }
217 | 
218 | 
219 | } }  // namespace rejit::internal
220 | 


--------------------------------------------------------------------------------
/src/assembler-base.h:
--------------------------------------------------------------------------------
  1 | // Copyright (c) 1994-2006 Sun Microsystems Inc.
  2 | // All Rights Reserved.
  3 | //
  4 | // Redistribution and use in source and binary forms, with or without
  5 | // modification, are permitted provided that the following conditions are
  6 | // met:
  7 | //
  8 | // - Redistributions of source code must retain the above copyright notice,
  9 | // this list of conditions and the following disclaimer.
 10 | //
 11 | // - Redistribution in binary form must reproduce the above copyright
 12 | // notice, this list of conditions and the following disclaimer in the
 13 | // documentation and/or other materials provided with the distribution.
 14 | //
 15 | // - Neither the name of Sun Microsystems or the names of contributors may
 16 | // be used to endorse or promote products derived from this software without
 17 | // specific prior written permission.
 18 | //
 19 | // THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
 20 | // IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
 21 | // THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
 22 | // PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
 23 | // CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
 24 | // EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
 25 | // PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
 26 | // PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
 27 | // LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
 28 | // NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
 29 | // SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 30 | 
 31 | // The original source code covered by the above license above has been
 32 | // modified significantly by Google Inc.
 33 | // Copyright 2012 the V8 project authors. All rights reserved.
 34 | 
 35 | #ifndef REJIT_ASSEMBLER_BASE_H_
 36 | #define REJIT_ASSEMBLER_BASE_H_
 37 | 
 38 | #include "globals.h"
 39 | #include "regexp.h"
 40 | 
 41 | #include <map>
 42 | 
 43 | namespace rejit {
 44 | namespace internal {
 45 | 
 46 | 
 47 | class AssemblerBase;
 48 | 
 49 | 
 50 | // -----------------------------------------------------------------------------
 51 | // Labels represent pc locations; they are typically jump or call targets.
 52 | // After declaration, a label can be freely used to denote known or (yet)
 53 | // unknown pc location. Assembler::bind() is used to bind a label to the
 54 | // current pc. A label can be bound only once.
 55 | 
 56 | class Label {
 57 |  public:
 58 |   enum Distance {
 59 |     kNear, kFar
 60 |   };
 61 | 
 62 |   inline Label() {
 63 |     Unuse();
 64 |     UnuseNear();
 65 |   }
 66 | 
 67 |   inline ~Label() {
 68 |     ASSERT(!is_linked());
 69 |     ASSERT(!is_near_linked());
 70 |   }
 71 | 
 72 |   inline void Unuse() { pos_ = 0; }
 73 |   inline void UnuseNear() { near_link_pos_ = 0; }
 74 | 
 75 |   inline bool is_bound() const { return pos_ <  0; }
 76 |   inline bool is_unused() const { return pos_ == 0 && near_link_pos_ == 0; }
 77 |   inline bool is_linked() const { return pos_ >  0; }
 78 |   inline bool is_near_linked() const { return near_link_pos_ > 0; }
 79 | 
 80 |   // Returns the position of bound or linked labels. Cannot be used
 81 |   // for unused labels.
 82 |   int pos() const;
 83 |   int near_link_pos() const { return near_link_pos_ - 1; }
 84 | 
 85 |  private:
 86 |   // pos_ encodes both the binding state (via its sign)
 87 |   // and the binding position (via its value) of a label.
 88 |   //
 89 |   // pos_ <  0  bound label, pos() returns the jump target position
 90 |   // pos_ == 0  unused label
 91 |   // pos_ >  0  linked label, pos() returns the last reference position
 92 |   int pos_;
 93 | 
 94 |   // Behaves like |pos_| in the "> 0" case, but for near jumps to this label.
 95 |   int near_link_pos_;
 96 | 
 97 |   void bind_to(int pos)  {
 98 |     pos_ = -pos - 1;
 99 |     ASSERT(is_bound());
100 |   }
101 |   void link_to(int pos, Distance distance = kFar) {
102 |     if (distance == kNear) {
103 |       near_link_pos_ = pos + 1;
104 |       ASSERT(is_near_linked());
105 |     } else {
106 |       pos_ = pos + 1;
107 |       ASSERT(is_linked());
108 |     }
109 |   }
110 | 
111 |   friend class Assembler;
112 |   friend class RegexpAssembler;
113 |   friend class Displacement;
114 |   friend class RegExpMacroAssemblerIrregexp;
115 | };
116 | 
117 | 
118 | // -----------------------------------------------------------------------------
119 | // Data relocation
120 | // Data may not fit the immediate fields of the instructions, and may not be
121 | // convenient to generate on the fly.
122 | // Code generation uses very little relocation and does not have as complex
123 | // requirements as v8, so we use a simpler relocation system.
124 | class RelocatedData {
125 |  public:
126 |   // The relocated value can be registered in an assembler that will take care
127 |   // of freeing it at the same time it is destroyed.
128 |   // The relocated value must be aligned according to the mask.
129 |   RelocatedData(char *buf, int buf_size, bool copy_buf,
130 |                 int alignment);
131 |   ~RelocatedData();
132 | 
133 |   inline int buffer_size() const { return buffer_size_; }
134 |   inline unsigned alignment() const { return alignment_; }
135 | 
136 |  private:
137 |   char *buffer_;
138 |   int buffer_size_;
139 |   bool own_buffer_;
140 |   int alignment_;
141 | 
142 |   friend class AssemblerBase;
143 | };
144 | 
145 | class RelocatedValue {
146 |  public:
147 |   RelocatedValue() :
148 |       data_(NULL), offset_(0) {}
149 | 
150 |   RelocatedValue(RelocatedData *data, unsigned offset) :
151 |       data_(data), offset_(offset) { ASSERT(offset_ >= 0); }
152 | 
153 |   inline RelocatedData *data() const { return data_; }
154 | 
155 |   bool operator<(const RelocatedValue& other) const {
156 |     return data_ < other.data_;
157 |   }
158 | 
159 |  private:
160 |   RelocatedData *data_;
161 |   // Offset from the base of the data.
162 |   int32_t offset_;
163 | 
164 |   friend class AssemblerBase;
165 |   friend class Assembler;
166 | };
167 | 
168 | 
169 | // -----------------------------------------------------------------------------
170 | // Platform independent assembler base class.
171 | 
172 | class AssemblerBase {
173 |   // TODO(rames): The AssemblerBase should manage the code generation buffer.
174 |  public:
175 |   explicit AssemblerBase(size_t min_buffer_size, size_t max_buffer_size,
176 |                          unsigned max_instr_size,
177 |                          void* buffer, int buffer_size);
178 |   ~AssemblerBase();
179 | 
180 |   // Code buffer management --------------------------------
181 |   VirtualMemory* GetCode();
182 | 
183 |   void GrowBuffer(bool force = false);
184 | 
185 |   inline int pc_offset() const { return static_cast<int>(pc_ - buffer_); }
186 | 
187 |   // Check if there is less than kGap bytes available in the buffer.
188 |   // If this is the case, we need to grow the buffer before emitting
189 |   // an instruction or relocation information.
190 |   inline bool buffer_overflow() const {
191 |     return pc_ >= buffer_ + buffer_size_ - max_instr_size_;
192 |   }
193 | 
194 |   // Get the number of bytes available in the buffer.
195 |   inline int available_space() const {
196 |     return static_cast<int>(buffer_ + buffer_size_ - pc_);
197 |   }
198 | 
199 |   byte byte_at(int pos)  { return buffer_[pos]; }
200 |   void set_byte_at(int pos, byte value) { buffer_[pos] = value; }
201 | 
202 | 
203 |   // Relocation --------------------------------------------
204 |   RelocatedData *NewRelocatedData(char *buf, size_t buf_size, bool copy_buf,
205 |                                   int alignment);
206 |   void EmitRelocData();
207 | 
208 |   void UseRelocatedData(RelocatedData *data);
209 |   void UseRelocatedValue(RelocatedValue reloc);
210 | 
211 | 
212 |  protected:
213 |   // Architecture specific values.
214 |   const int min_buffer_size_;
215 |   const int max_buffer_size_;
216 |   // Used to check that the buffer is big enough before assembling an
217 |   // instruction.
218 |   const int max_instr_size_;
219 | 
220 |   // Code buffer:
221 |   // The buffer into which code is generated.
222 |   byte* buffer_;
223 |   int buffer_size_;
224 |   // True if the assembler owns the buffer, false if buffer is external.
225 |   bool own_buffer_;
226 |   // code generation
227 |   byte* pc_;  // the program counter; moves forward
228 | 
229 |   vector<RelocatedData*> reloc_data_owned_;
230 |   // The relocated data used by this assembler and the offset at which the data
231 |   // has been emitted in the buffer.
232 |   map<RelocatedData*, int> reloc_data_location_;
233 |   // The relocated values and the locations (offsets) at which they are used.
234 |   map<RelocatedValue, int> reloc_values_usage_location_;
235 | };
236 | 
237 | } }  // namespace rejit::internal
238 | 
239 | #endif  // REJIT_ASSEMBLER_BASE_H_
240 | 


--------------------------------------------------------------------------------
/src/assembler.h:
--------------------------------------------------------------------------------
 1 | // Copyright (C) 2013 Alexandre Rames <alexandre@coreperf.com>
 2 | // rejit is free software: you can redistribute it and/or modify
 3 | // it under the terms of the GNU General Public License as published by
 4 | // the Free Software Foundation, either version 3 of the License, or
 5 | // (at your option) any later version.
 6 | //
 7 | // This program is distributed in the hope that it will be useful,
 8 | // but WITHOUT ANY WARRANTY; without even the implied warranty of
 9 | // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
10 | // GNU General Public License for more details.
11 | //
12 | // You should have received a copy of the GNU General Public License
13 | // along with this program.  If not, see <http://www.gnu.org/licenses/>.
14 | 
15 | #ifndef REJIT_ASSEMBLER_H_
16 | #define REJIT_ASSEMBLER_H_
17 | 
18 | #include "globals.h"
19 | 
20 | namespace rejit {
21 | namespace internal {
22 | 
23 | 
24 | #ifdef REJIT_TARGET_ARCH_X64
25 | #include "x64/assembler-x64.h"
26 | #include "x64/assembler-x64-inl.h"
27 | #else
28 | #error Unkown target architecture.
29 | #endif
30 | 
31 | } }  // namespace rejit::internal
32 | 
33 | #endif  // REJIT_ASSEMBLER_H_
34 | 


--------------------------------------------------------------------------------
/src/checks.cc:
--------------------------------------------------------------------------------
 1 | // Copyright (C) 2013 Alexandre Rames <alexandre@coreperf.com>
 2 | // rejit is free software: you can redistribute it and/or modify
 3 | // it under the terms of the GNU General Public License as published by
 4 | // the Free Software Foundation, either version 3 of the License, or
 5 | // (at your option) any later version.
 6 | //
 7 | // This program is distributed in the hope that it will be useful,
 8 | // but WITHOUT ANY WARRANTY; without even the implied warranty of
 9 | // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
10 | // GNU General Public License for more details.
11 | //
12 | // You should have received a copy of the GNU General Public License
13 | // along with this program.  If not, see <http://www.gnu.org/licenses/>.
14 | 
15 | #include "checks.h"
16 | #include <stdio.h>
17 | #include <stdlib.h>
18 | 
19 | void rejit_fatal(const char* file, int line, const char* format, ...) {
20 |   printf("ERROR: in %s, line %d: ", file, line);
21 |   va_list args;
22 |   va_start(args, format);
23 |   printf(format, args);
24 |   va_end(args);
25 |   abort();
26 | }
27 | 


--------------------------------------------------------------------------------
/src/checks.h:
--------------------------------------------------------------------------------
 1 | // Copyright (C) 2013 Alexandre Rames <alexandre@coreperf.com>
 2 | // rejit is free software: you can redistribute it and/or modify
 3 | // it under the terms of the GNU General Public License as published by
 4 | // the Free Software Foundation, either version 3 of the License, or
 5 | // (at your option) any later version.
 6 | //
 7 | // This program is distributed in the hope that it will be useful,
 8 | // but WITHOUT ANY WARRANTY; without even the implied warranty of
 9 | // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
10 | // GNU General Public License for more details.
11 | //
12 | // You should have received a copy of the GNU General Public License
13 | // along with this program.  If not, see <http://www.gnu.org/licenses/>.
14 | 
15 | #ifndef REJIT_CHECKS_H
16 | #define REJIT_CHECKS_H
17 | 
18 | #include <assert.h>
19 | #include <stdarg.h>
20 | 
21 | void rejit_fatal(const char* file, int line, const char* format, ...);
22 | 
23 | #define STATIC_ASSERT(condition) static_assert((condition), "static_assert")
24 | #define ALWAYS_ASSERT(condition) assert((condition))
25 | #define FATAL(s) rejit_fatal(__FILE__, __LINE__, "FATAL: %s\n", s)
26 | 
27 | #ifdef DEBUG
28 | #define ASSERT(condition) assert((condition))
29 | #define UNREACHABLE() rejit_fatal(__FILE__, __LINE__, "UNREACHABLE\n")
30 | #define UNIMPLEMENTED() rejit_fatal(__FILE__, __LINE__, "UNIMPLEMENTED\n")
31 | #else
32 | #define ASSERT(condition)
33 | #define UNREACHABLE() rejit_fatal("", 0, "UNREACHABLE\n")
34 | #define UNIMPLEMENTED() rejit_fatal("", 0, "UNIMPLEMENTED\n")
35 | #endif
36 | 
37 | #endif  // REJIT_CHECKS_H
38 | 


--------------------------------------------------------------------------------
/src/config.h:
--------------------------------------------------------------------------------
 1 | // Copyright (C) 2013 Alexandre Rames <alexandre@coreperf.com>
 2 | 
 3 | // rejit is free software: you can redistribute it and/or modify
 4 | // it under the terms of the GNU General Public License as published by
 5 | // the Free Software Foundation, either version 3 of the License, or
 6 | // (at your option) any later version.
 7 | //
 8 | // This program is distributed in the hope that it will be useful,
 9 | // but WITHOUT ANY WARRANTY; without even the implied warranty of
10 | // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
11 | // GNU General Public License for more details.
12 | //
13 | // You should have received a copy of the GNU General Public License
14 | // along with this program.  If not, see <http://www.gnu.org/licenses/>.
15 | 
16 | #ifndef REJIT_CONFIG_H_
17 | #define REJIT_CONFIG_H_
18 | 
19 | // Uncomment/comment options below to enable/disable them.
20 | 
21 | // The following patterns are not defined in the ISO-IEC 9945, but are widely
22 | // supported and used.
23 | // Allow support for them.
24 | //  \d   : Digit character; equivalent to [0-9].
25 | //  \D   : Non digit character; equivalent to [^0-9].
26 | //  \n
27 | //  \r
28 | //  \s   : Whitespace character (space or tab).
29 | //  \S   : Non whitespace character.
30 | //  \t   : Tab character.
31 | //  \xAB : Character with hexadecimal code 0xAB.
32 | // TODO: Add support for:
33 | //  \<, \>
34 | #define ENABLE_COMMON_ESCAPED_PATTERNS
35 | 
36 | #endif
37 | 


--------------------------------------------------------------------------------
/src/cpu.h:
--------------------------------------------------------------------------------
  1 | // Copyright 2012 the V8 project authors. All rights reserved.
  2 | // Redistribution and use in source and binary forms, with or without
  3 | // modification, are permitted provided that the following conditions are
  4 | // met:
  5 | //
  6 | //     * Redistributions of source code must retain the above copyright
  7 | //       notice, this list of conditions and the following disclaimer.
  8 | //     * Redistributions in binary form must reproduce the above
  9 | //       copyright notice, this list of conditions and the following
 10 | //       disclaimer in the documentation and/or other materials provided
 11 | //       with the distribution.
 12 | //     * Neither the name of Google Inc. nor the names of its
 13 | //       contributors may be used to endorse or promote products derived
 14 | //       from this software without specific prior written permission.
 15 | //
 16 | // THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
 17 | // "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
 18 | // LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
 19 | // A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
 20 | // OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
 21 | // SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
 22 | // LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
 23 | // DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
 24 | // THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
 25 | // (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
 26 | // OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 27 | 
 28 | // Copyright (C) 2013 Alexandre Rames <alexandre@coreperf.com>
 29 | // rejit is free software: you can redistribute it and/or modify
 30 | // it under the terms of the GNU General Public License as published by
 31 | // the Free Software Foundation, either version 3 of the License, or
 32 | // (at your option) any later version.
 33 | //
 34 | // This program is distributed in the hope that it will be useful,
 35 | // but WITHOUT ANY WARRANTY; without even the implied warranty of
 36 | // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 37 | // GNU General Public License for more details.
 38 | //
 39 | // You should have received a copy of the GNU General Public License
 40 | // along with this program.  If not, see <http://www.gnu.org/licenses/>.
 41 | 
 42 | 
 43 | #ifndef REJIT_CPU_H_
 44 | #define REJIT_CPU_H_
 45 | 
 46 | // TODO(rames): We should detect that at runtime.
 47 | #define MAX_PARALLEL_CHARS 4
 48 | static const int kMaxParallelChars = 4;
 49 | static const int kMaxParallelCharsLog2 = 2;
 50 | 
 51 | namespace rejit {
 52 | namespace internal {
 53 | 
 54 | // This is from v8/src/v8globals.h.
 55 | //
 56 | // Feature flags bit positions. They are mostly based on the CPUID spec.
 57 | // (We assign CPUID itself to one of the currently reserved bits --
 58 | // feel free to change this if needed.)
 59 | // On X86/X64, values below 32 are bits in EDX, values above 32 are bits in ECX.
 60 | enum CpuFeature {
 61 |   SSE4_2 = 32 + 20,  // x86
 62 |   SSE4_1 = 32 + 19,  // x86
 63 |   SSE3 = 32 + 0,     // x86
 64 |   SSE2 = 26,   // x86
 65 |   CMOV = 15,   // x86
 66 |   RDTSC = 4,   // x86
 67 |   CPUID = 10,  // x86
 68 |   VFP3 = 1,    // ARM
 69 |   ARMv7 = 2,   // ARM
 70 |   VFP2 = 3,    // ARM
 71 |   SAHF = 0,    // x86
 72 |   FPU = 1
 73 | };    // MIPS
 74 | 
 75 | // ----------------------------------------------------------------------------
 76 | // CPU
 77 | //
 78 | // This class has static methods for the architecture specific functions. Add
 79 | // methods here to cope with differences between the supported architectures.
 80 | //
 81 | // For each architecture the file cpu_<arch>.cc contains the implementation of
 82 | // these functions.
 83 | 
 84 | class CPU {
 85 |  public:
 86 |   // Initializes the cpu architecture support. Called once at VM startup.
 87 |   static void SetUp();
 88 | 
 89 |   static bool SupportsCrankshaft();
 90 | 
 91 |   // Flush instruction cache.
 92 |   static void FlushICache(void* start, size_t size);
 93 | 
 94 |   // Try to activate a system level debugger.
 95 |   static void DebugBreak();
 96 | };
 97 | 
 98 | } }  // namespace rejit::internal
 99 | 
100 | 
101 | #endif  // REJIT_CPU_H_
102 | 
103 | 


--------------------------------------------------------------------------------
/src/flags.cc:
--------------------------------------------------------------------------------
 1 | // Copyright (C) 2013 Alexandre Rames <alexandre@coreperf.com>
 2 | // rejit is free software: you can redistribute it and/or modify
 3 | // it under the terms of the GNU General Public License as published by
 4 | // the Free Software Foundation, either version 3 of the License, or
 5 | // (at your option) any later version.
 6 | //
 7 | // This program is distributed in the hope that it will be useful,
 8 | // but WITHOUT ANY WARRANTY; without even the implied warranty of
 9 | // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
10 | // GNU General Public License for more details.
11 | //
12 | // You should have received a copy of the GNU General Public License
13 | // along with this program.  If not, see <http://www.gnu.org/licenses/>.
14 | 
15 | #include "flags.h"
16 | 
17 | #if defined(DEBUG) && defined(MOD_FLAGS)
18 | #define DEFINE_FLAG(name, r, debug_def) bool FLAG_##name = debug_def;
19 | #elif defined(MOD_FLAGS)
20 | #define DEFINE_FLAG(name, release_def, d) bool FLAG_##name = release_def;
21 | #else
22 | #define DEFINE_FLAG(name, r, d)
23 | #endif
24 | REJIT_FLAGS_LIST(DEFINE_FLAG)
25 | #undef DEFINE_FLAG
26 | 


--------------------------------------------------------------------------------
/src/flags.h:
--------------------------------------------------------------------------------
 1 | // Copyright (C) 2013 Alexandre Rames <alexandre@coreperf.com>
 2 | // rejit is free software: you can redistribute it and/or modify
 3 | // it under the terms of the GNU General Public License as published by
 4 | // the Free Software Foundation, either version 3 of the License, or
 5 | // (at your option) any later version.
 6 | //
 7 | // This program is distributed in the hope that it will be useful,
 8 | // but WITHOUT ANY WARRANTY; without even the implied warranty of
 9 | // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
10 | // GNU General Public License for more details.
11 | //
12 | // You should have received a copy of the GNU General Public License
13 | // along with this program.  If not, see <http://www.gnu.org/licenses/>.
14 | 
15 | #ifndef REJIT_FLAGS_H_
16 | #define REJIT_FLAGS_H_
17 | 
18 | // This is a centralised definition of the flags used by rejit.
19 | // This allows to easily use macros to operate on all the flags at once.
20 | // Note that defined macros are required to take 3 arguments, but can ignore
21 | // some of them (often the debug and release mode default values).
22 | // See for example the DECLARE_FLAG macro.
23 | 
24 | #define REJIT_PRINT_FLAGS_LIST(M)                                              \
25 | /* flag_name             , release , debug mode default value */               \
26 | M( print_ff_elements     , false   , false )                                   \
27 | /* Print the list of regexps that the codegen will generate code for. */       \
28 | M( print_re_list         , false   , false )                                   \
29 | /* Print the regexp tree after parsing. */                                     \
30 | M( print_re_tree         , false   , false )                                   \
31 | /* Print info about the state ring. */                                         \
32 | M( print_state_ring_info , false   , false )                                   \
33 | /* Display information about reduction of fast-forward elements. */            \
34 | M( print_ff_reduce       , false   , false )                                   \
35 | 
36 | #define REJIT_FLAGS_LIST(M)                                                    \
37 | /* flag_name             , release , debug mode default value */               \
38 | /* Emit extra code for debugging purposes. */                                  \
39 | M( emit_debug_code       , false   , true  )                                   \
40 | /* Show the ff elements chosen by the fast forward mechanism. */               \
41 | /* For kMatchAll, print every match when registered. */                        \
42 | M( trace_match_all       , false   , false )                                   \
43 | /* Trace repetitions handling at parse time. */                                \
44 | M( trace_repetitions     , false   , false )                                   \
45 | /* Use the fast-forwarding mechanisms. */                                      \
46 | M( use_fast_forward      , true    , true  )                                   \
47 | /* Fast-forward early to improve the scanning speed when no matches appear. */ \
48 | M( use_fast_forward_early, true    , true  )                                   \
49 | /* Use / trace reduction of fast-forward elements (substring extraction). */   \
50 | M( use_ff_reduce         , true    , true  )                                   \
51 | /* Use parser level optimizations. */                                          \
52 | M( use_parser_opt        , true    , true  )                                   \
53 | /* Dump generated code. */                                                     \
54 | M( dump_code             , false   , false )                                   \
55 | REJIT_PRINT_FLAGS_LIST(M)
56 | 
57 | // Declare all the flags.
58 | #if defined(DEBUG) || defined(MOD_FLAGS)
59 | #define DECLARE_FLAG(name, r, d) extern bool FLAG_##name;
60 | #else
61 | #define DECLARE_FLAG(name, release_def, x) \
62 |   static const bool FLAG_##name = release_def;
63 | #endif
64 | REJIT_FLAGS_LIST(DECLARE_FLAG)
65 | #undef DECLARE_FLAG
66 | 
67 | #ifdef MOD_FLAGS
68 | #define SET_FLAG(name, val) FLAG_##name = val
69 | #else
70 | #define SET_FLAG(name, val)
71 | #endif
72 | 
73 | // This removes early exits to allow benchmarking.
74 | // If set to false, searching for matches with a NULL match results will
75 | // bailout early. If set to true, code will actually search for
76 | // matches but simply not update the (NULL) match results.
77 | #ifdef BENCHTEST
78 | #define FLAG_benchtest true
79 | #else
80 | #define FLAG_benchtest false
81 | #endif
82 | 
83 | #endif  // REJIT_FLAGS_H_
84 | 


--------------------------------------------------------------------------------
/src/globals.h:
--------------------------------------------------------------------------------
 1 | // Copyright 2012 the V8 project authors. All rights reserved.
 2 | // Redistribution and use in source and binary forms, with or without
 3 | // modification, are permitted provided that the following conditions are
 4 | // met:
 5 | //
 6 | //     * Redistributions of source code must retain the above copyright
 7 | //       notice, this list of conditions and the following disclaimer.
 8 | //     * Redistributions in binary form must reproduce the above
 9 | //       copyright notice, this list of conditions and the following
10 | //       disclaimer in the documentation and/or other materials provided
11 | //       with the distribution.
12 | //     * Neither the name of Google Inc. nor the names of its
13 | //       contributors may be used to endorse or promote products derived
14 | //       from this software without specific prior written permission.
15 | //
16 | // THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
17 | // "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
18 | // LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
19 | // A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
20 | // OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
21 | // SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
22 | // LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
23 | // DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
24 | // THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
25 | // (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
26 | // OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
27 | 
28 | #ifndef REJIT_GLOBALS_H_
29 | #define REJIT_GLOBALS_H_
30 | 
31 | #include <stdint.h>
32 | #include <stddef.h>
33 | #include <string.h>
34 | #include <vector>
35 | #include <iostream>
36 | #include <algorithm>
37 | 
38 | #include "rejit.h"
39 | #include "config.h"
40 | #include "checks.h"
41 | #include "flags.h"
42 | #include "cpu.h"
43 | 
44 | using namespace std;  // NOLINT
45 | 
46 | namespace rejit {
47 | namespace internal {
48 | 
49 | const int KB = 1024;
50 | const int MB = KB * KB;
51 | 
52 | const uint32_t kMaxUInt = 0xffffffff;
53 | const uint64_t kMaxUInt64 = 0xffffffffffffffffULL;
54 | 
55 | const int kPointerSize = sizeof(void*);  // NOLINT
56 | const int kPointerSizeLog2 = 3;
57 | const int kBitsPerByte = 8;
58 | const int kBitsPerPointer = kPointerSize * kBitsPerByte;
59 | const int kCharSize = sizeof(char);  // NOLINT
60 | const int kBitsPerChar = 8;
61 | const int kCharsPerPointer = kPointerSize / kCharSize;
62 | 
63 | typedef unsigned char byte;
64 | typedef byte* Address;
65 | 
66 | // The signedness of char is implementation defined.
67 | // See C++ standard 3.9.1 Fundamental types (1.)
68 | typedef unsigned char uchar;
69 | 
70 | 
71 | #define DISALLOW_COPY_AND_ASSIGN(Type) \
72 |   Type(const Type&);                   \
73 |   void operator=(const Type&)
74 | 
75 | // Use this to avoid unused variable warnings.
76 | template <typename T>
77 | void USE(T) {}
78 | 
79 | // FUNCTION_ADDR(f) gets the address of a C function f.
80 | #define FUNCTION_ADDR(f)                                        \
81 |   (reinterpret_cast<rejit::internal::Address>(reinterpret_cast<intptr_t>(f)))
82 | 
83 | // FUNCTION_CAST<F>(addr) casts an address into a function
84 | // of type F. Used to invoke generated code from within C.
85 | template <typename F>
86 | F FUNCTION_CAST(Address addr) {
87 |   return reinterpret_cast<F>(reinterpret_cast<intptr_t>(addr));
88 | }
89 | 
90 | } }  // namespace rejit::internal
91 | 
92 | #endif  // REJIT_GLOBALS_H_
93 | 
94 | 


--------------------------------------------------------------------------------
/src/macro-assembler-base.h:
--------------------------------------------------------------------------------
 1 | // Copyright (C) 2013 Alexandre Rames <alexandre@coreperf.com>
 2 | // rejit is free software: you can redistribute it and/or modify
 3 | // it under the terms of the GNU General Public License as published by
 4 | // the Free Software Foundation, either version 3 of the License, or
 5 | // (at your option) any later version.
 6 | //
 7 | // This program is distributed in the hope that it will be useful,
 8 | // but WITHOUT ANY WARRANTY; without even the implied warranty of
 9 | // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
10 | // GNU General Public License for more details.
11 | //
12 | // You should have received a copy of the GNU General Public License
13 | // along with this program.  If not, see <http://www.gnu.org/licenses/>.
14 | 
15 | #include "assembler.h"
16 | 
17 | namespace rejit {
18 | namespace internal {
19 | 
20 | class MacroAssemblerBase : public Assembler {
21 |  public:
22 |   MacroAssemblerBase(void* buffer, int buffer_size)
23 |     : Assembler(buffer, buffer_size), char_size_(kCharSize) {}
24 | 
25 |   inline unsigned char_size() const { return char_size_; }
26 |   inline void set_char_size(unsigned size) { char_size_ = size; }
27 | 
28 |  private:
29 |   unsigned char_size_;
30 | };
31 | 
32 | enum Direction {
33 |   kForward,
34 |   kBackward
35 | };
36 | 
37 | 
38 | } }  // namespace rejit::internal
39 | 
40 | 


--------------------------------------------------------------------------------
/src/macro-assembler.h:
--------------------------------------------------------------------------------
 1 | // Copyright (C) 2013 Alexandre Rames <alexandre@coreperf.com>
 2 | // rejit is free software: you can redistribute it and/or modify
 3 | // it under the terms of the GNU General Public License as published by
 4 | // the Free Software Foundation, either version 3 of the License, or
 5 | // (at your option) any later version.
 6 | //
 7 | // This program is distributed in the hope that it will be useful,
 8 | // but WITHOUT ANY WARRANTY; without even the implied warranty of
 9 | // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
10 | // GNU General Public License for more details.
11 | //
12 | // You should have received a copy of the GNU General Public License
13 | // along with this program.  If not, see <http://www.gnu.org/licenses/>.
14 | 
15 | #ifndef MACRO_ASSEMBLER_H_
16 | #define MACRO_ASSEMBLER_H_
17 | 
18 | #ifdef REJIT_TARGET_ARCH_X64
19 | #include "x64/macro-assembler-x64.h"
20 | #include "x64/macro-assembler-x64-inl.h"
21 | #else
22 | #error Unkown target architecture.
23 | #endif
24 | 
25 | #endif  // MACRO_ASSEMBLER_H_
26 | 


--------------------------------------------------------------------------------
/src/memory.h:
--------------------------------------------------------------------------------
 1 | // Copyright 2011 the V8 project authors. All rights reserved.
 2 | // Redistribution and use in source and binary forms, with or without
 3 | // modification, are permitted provided that the following conditions are
 4 | // met:
 5 | //
 6 | //     * Redistributions of source code must retain the above copyright
 7 | //       notice, this list of conditions and the following disclaimer.
 8 | //     * Redistributions in binary form must reproduce the above
 9 | //       copyright notice, this list of conditions and the following
10 | //       disclaimer in the documentation and/or other materials provided
11 | //       with the distribution.
12 | //     * Neither the name of Google Inc. nor the names of its
13 | //       contributors may be used to endorse or promote products derived
14 | //       from this software without specific prior written permission.
15 | //
16 | // THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
17 | // "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
18 | // LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
19 | // A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
20 | // OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
21 | // SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
22 | // LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
23 | // DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
24 | // THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
25 | // (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
26 | // OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
27 | 
28 | #ifndef REJIT_MEMORY_H_
29 | #define REJIT_MEMORY_H_
30 | 
31 | namespace rejit {
32 | namespace internal {
33 | 
34 | // Memory provides an interface to 'raw' memory. It encapsulates the casts
35 | // that typically are needed when incompatible pointer types are used.
36 | 
37 | class Memory {
38 |  public:
39 |   static uint8_t& uint8_at(Address addr) {
40 |     return *reinterpret_cast<uint8_t*>(addr);
41 |   }
42 | 
43 |   static uint16_t& uint16_at(Address addr)  {
44 |     return *reinterpret_cast<uint16_t*>(addr);
45 |   }
46 | 
47 |   static uint32_t& uint32_at(Address addr)  {
48 |     return *reinterpret_cast<uint32_t*>(addr);
49 |   }
50 | 
51 |   static int32_t& int32_at(Address addr)  {
52 |     return *reinterpret_cast<int32_t*>(addr);
53 |   }
54 | 
55 |   static uint64_t& uint64_at(Address addr)  {
56 |     return *reinterpret_cast<uint64_t*>(addr);
57 |   }
58 | 
59 |   static int& int_at(Address addr)  {
60 |     return *reinterpret_cast<int*>(addr);
61 |   }
62 | 
63 |   static unsigned& unsigned_at(Address addr) {
64 |     return *reinterpret_cast<unsigned*>(addr);
65 |   }
66 | 
67 |   static double& double_at(Address addr)  {
68 |     return *reinterpret_cast<double*>(addr);
69 |   }
70 | 
71 |   static Address& Address_at(Address addr)  {
72 |     return *reinterpret_cast<Address*>(addr);
73 |   }
74 | };
75 | 
76 | } }  // namespace rejit::internal
77 | 
78 | #endif  // REJIT_MEMORY_H_
79 | 


--------------------------------------------------------------------------------
/src/parser.h:
--------------------------------------------------------------------------------
  1 | // Copyright (C) 2013 Alexandre Rames <alexandre@coreperf.com>
  2 | // rejit is free software: you can redistribute it and/or modify
  3 | // it under the terms of the GNU General Public License as published by
  4 | // the Free Software Foundation, either version 3 of the License, or
  5 | // (at your option) any later version.
  6 | //
  7 | // This program is distributed in the hope that it will be useful,
  8 | // but WITHOUT ANY WARRANTY; without even the implied warranty of
  9 | // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 10 | // GNU General Public License for more details.
 11 | //
 12 | // You should have received a copy of the GNU General Public License
 13 | // along with this program.  If not, see <http://www.gnu.org/licenses/>.
 14 | 
 15 | #ifndef REJIT_PARSER_H_
 16 | #define REJIT_PARSER_H_
 17 | 
 18 | #include "regexp.h"
 19 | 
 20 | namespace rejit {
 21 | namespace internal {
 22 | 
 23 | // TODO: Full support for ERE. Expose BRE support. More syntaxes
 24 | enum Syntax {
 25 |   BRE,
 26 |   ERE
 27 | };
 28 | 
 29 | // TODO: The indexer and ff_finder should be merged into the parser.
 30 | 
 31 | // Regular expression parser.
 32 | class Parser {
 33 |  public:
 34 |   Parser() {}
 35 | 
 36 |   // Top level function to parse a regular expression.
 37 |   inline Status Parse(Syntax syntax, RegexpInfo* rinfo, const char* regexp) {
 38 |     syntax_ = syntax;
 39 |     regexp_info_ = rinfo;
 40 |     regexp_string_ = regexp;
 41 |     status_ = RejitSuccess;
 42 |     stack_.clear();
 43 |     switch(syntax) {
 44 |       case BRE:
 45 |         return ParseBRE(rinfo, regexp);
 46 |       case ERE:
 47 |         return ParseERE(rinfo, regexp);
 48 |       default:
 49 |         UNREACHABLE();
 50 |         return RejitSuccess;
 51 |     }
 52 |   }
 53 | 
 54 |   Status ParseERE(RegexpInfo* rinfo, const char* regexp);
 55 |   Status ParseBRE(RegexpInfo* rinfo, const char* regexp);
 56 | 
 57 |   // Parsing functions that can fail can return a negative progress to indicate
 58 |   // a failure.
 59 |   int ParseCurlyBrackets(const char* left_curly_bracket);
 60 |   int ParseBrackets(const char* left_bracket);
 61 | 
 62 |   // Do/Push functions -----------------------------------------------
 63 |   // 'Push' functions' main purpose is to push something on the stack.
 64 |   // 'Do' functions execute more complex operations.
 65 |   // TODO: Give more details for each function.
 66 |   void PushChar(char c, bool append_to_mc_tos = true);
 67 |   void PushChar(const char* char_address);
 68 |   void PushPeriod();
 69 |   void PushAlternateBar();
 70 |   void PushLeftParenthesis();
 71 |   void PushAsterisk();
 72 |   void PushPlus();
 73 |   void PushQuestionMark();
 74 | 
 75 |   void DoRightParenthesis();
 76 |   void DoAlternateBar();
 77 | 
 78 |   void DoConcatenation();
 79 |   void DoAlternation();
 80 | 
 81 |   void DoFinish();
 82 | 
 83 | 
 84 | 
 85 |   // Stack access helpers --------------------------------------------
 86 |   // tos = top of stack.
 87 |   inline Regexp* tos() { return stack()->size() ? stack()->back() : NULL; }
 88 | 
 89 |   void PushRegexp(Regexp* regexp) {
 90 |     stack()->push_back(regexp);
 91 |   }
 92 | 
 93 |   Regexp* PopRegexp() {
 94 |     Regexp* r = stack()->back();
 95 |     stack()->pop_back();
 96 |     return r;
 97 |   }
 98 | 
 99 | 
100 |   bool IsRetroactiveChar(char c) {
101 |     if (c == '*') return true;
102 |     if (c == '{') return true;
103 |     return false;
104 |   }
105 | 
106 |   void PrintStack() {
107 |     cout << "---------------------------" << endl;
108 |     vector<Regexp*>::iterator it;
109 |     for (it = stack()->begin(); it < stack()->end(); it++) {
110 |       cout << **it << endl;
111 |     }
112 |   }
113 | 
114 |   // Helpers ---------------------------------------------------------
115 |   uint32_t ParseIntegerAt(const char* pos, char** end);
116 | 
117 |   // Error signaling -------------------------------------------------
118 |   Status ParseError(const char* pos, const char* format, ...);
119 | 
120 |   Status Unexpected(const char* pos);
121 | 
122 |   inline Status Unexpected(unsigned index) {
123 |     return Unexpected(regexp_string_ + index);
124 |   }
125 | 
126 |   Status Expected(const char* pos, const char *expected);
127 | 
128 |   inline Status Expected(unsigned index, const char *expected) {
129 |     return Expected(regexp_string_ + index, expected);
130 |   }
131 | 
132 |   Status Expect(const char *c, const char* expected);
133 | 
134 | 
135 |   RegexpInfo* regexp_info() { return regexp_info_; }
136 |   vector<Regexp*>* stack() { return &stack_; }
137 | 
138 |  private:
139 |   RegexpInfo* regexp_info_;
140 |   const char* regexp_string_;
141 |   uint64_t index_;
142 |   Syntax syntax_;
143 |   Status status_;
144 |   vector<Regexp*> stack_;
145 | };
146 | 
147 | } }  // namespace rejit::internal
148 | 
149 | #endif  // REJIT_PARSER_H_
150 | 
151 | 


--------------------------------------------------------------------------------
/src/platform/platform-linux.cc:
--------------------------------------------------------------------------------
  1 | // Copyright 2012 the V8 project authors. All rights reserved.
  2 | // Redistribution and use in source and binary forms, with or without
  3 | // modification, are permitted provided that the following conditions are
  4 | // met:
  5 | //
  6 | //     * Redistributions of source code must retain the above copyright
  7 | //       notice, this list of conditions and the following disclaimer.
  8 | //     * Redistributions in binary form must reproduce the above
  9 | //       copyright notice, this list of conditions and the following
 10 | //       disclaimer in the documentation and/or other materials provided
 11 | //       with the distribution.
 12 | //     * Neither the name of Google Inc. nor the names of its
 13 | //       contributors may be used to endorse or promote products derived
 14 | //       from this software without specific prior written permission.
 15 | //
 16 | // THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
 17 | // "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
 18 | // LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
 19 | // A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
 20 | // OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
 21 | // SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
 22 | // LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
 23 | // DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
 24 | // THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
 25 | // (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
 26 | // OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 27 | 
 28 | // Platform specific code for POSIX goes here. This is not a platform on its
 29 | // own but contains the parts which are the same across POSIX platforms Linux,
 30 | // Mac OS, FreeBSD and OpenBSD.
 31 | 
 32 | #include "platform-posix.h"
 33 | 
 34 | #include <unistd.h>
 35 | #include <errno.h>
 36 | #include <time.h>
 37 | 
 38 | #include <sys/mman.h>
 39 | #include <sys/socket.h>
 40 | #include <sys/resource.h>
 41 | #include <sys/time.h>
 42 | #include <sys/types.h>
 43 | #include <sys/stat.h>
 44 | 
 45 | #include <arpa/inet.h>
 46 | #include <netinet/in.h>
 47 | #include <netdb.h>
 48 | 
 49 | #include "platform.h"
 50 | #include "utils.h"
 51 | 
 52 | namespace rejit {
 53 | namespace internal {
 54 | 
 55 | size_t OS::AllocateAlignment() {
 56 |   return sysconf(_SC_PAGESIZE);
 57 | }
 58 | 
 59 | 
 60 | uint64_t OS::CpuFeaturesImpliedByPlatform() {
 61 |   return 0;  // Linux runs on anything.
 62 | }
 63 | 
 64 | 
 65 | void OS::Free(void* address, const size_t size) {
 66 |   // TODO(1240712): munmap has a return value which is ignored here.
 67 |   int result = munmap(address, size);
 68 |   USE(result);
 69 |   ASSERT(result == 0);
 70 | }
 71 | 
 72 | 
 73 | intptr_t OS::CommitPageSize() {
 74 |   static intptr_t page_size = getpagesize();
 75 |   return page_size;
 76 | }
 77 | 
 78 | 
 79 | // Get rid of writable permission on code allocations.
 80 | void OS::ProtectCode(void* address, const size_t size) {
 81 |   mprotect(address, size, PROT_READ | PROT_EXEC);
 82 | }
 83 | 
 84 | 
 85 | // Create guard pages.
 86 | void OS::Guard(void* address, const size_t size) {
 87 |   mprotect(address, size, PROT_NONE);
 88 | }
 89 | void* OS::GetRandomMmapAddr() {
 90 |   // TODO(rames): Implement this.
 91 |   // UNIMPLEMENTED();
 92 |   return NULL;
 93 | }
 94 | 
 95 | 
 96 | // Constants used for mmap.
 97 | static const int kMmapFd = -1;
 98 | static const int kMmapFdOffset = 0;
 99 | 
100 | VirtualMemory::VirtualMemory() : address_(NULL), size_(0) { }
101 | 
102 | VirtualMemory::VirtualMemory(size_t size) {
103 |   address_ = ReserveRegion(size);
104 |   size_ = size;
105 | }
106 | 
107 | 
108 | VirtualMemory::VirtualMemory(size_t size, size_t alignment)
109 |     : address_(NULL), size_(0) {
110 |   ASSERT(IsAligned(alignment, static_cast<intptr_t>(OS::AllocateAlignment())));
111 |   size_t request_size = RoundUp(size + alignment,
112 |                                 static_cast<intptr_t>(OS::AllocateAlignment()));
113 |   void* reservation = mmap(OS::GetRandomMmapAddr(),
114 |                            request_size,
115 |                            PROT_NONE,
116 |                            MAP_PRIVATE | MAP_ANONYMOUS | MAP_NORESERVE,
117 |                            kMmapFd,
118 |                            kMmapFdOffset);
119 |   if (reservation == MAP_FAILED) return;
120 | 
121 |   Address base = static_cast<Address>(reservation);
122 |   Address aligned_base = RoundUp(base, alignment);
123 |   ASSERT(base <= aligned_base);
124 | 
125 |   // Unmap extra memory reserved before and after the desired block.
126 |   if (aligned_base != base) {
127 |     size_t prefix_size = static_cast<size_t>(aligned_base - base);
128 |     OS::Free(base, prefix_size);
129 |     request_size -= prefix_size;
130 |   }
131 | 
132 |   size_t aligned_size = RoundUp(size, OS::AllocateAlignment());
133 |   ASSERT(aligned_size <= request_size);
134 | 
135 |   if (aligned_size != request_size) {
136 |     size_t suffix_size = request_size - aligned_size;
137 |     OS::Free(aligned_base + aligned_size, suffix_size);
138 |     request_size -= suffix_size;
139 |   }
140 | 
141 |   ASSERT(aligned_size == request_size);
142 | 
143 |   address_ = static_cast<void*>(aligned_base);
144 |   size_ = aligned_size;
145 | }
146 | 
147 | 
148 | VirtualMemory::~VirtualMemory() {
149 |   if (IsReserved()) {
150 |     bool result = ReleaseRegion(address(), size());
151 |     ASSERT(result);
152 |     USE(result);
153 |   }
154 | }
155 | 
156 | 
157 | bool VirtualMemory::IsReserved() {
158 |   return address_ != NULL;
159 | }
160 | 
161 | 
162 | void VirtualMemory::Reset() {
163 |   address_ = NULL;
164 |   size_ = 0;
165 | }
166 | 
167 | 
168 | bool VirtualMemory::Commit(void* address, size_t size, bool is_executable) {
169 |   return CommitRegion(address, size, is_executable);
170 | }
171 | 
172 | 
173 | bool VirtualMemory::Uncommit(void* address, size_t size) {
174 |   return UncommitRegion(address, size);
175 | }
176 | 
177 | 
178 | bool VirtualMemory::Guard(void* address) {
179 |   OS::Guard(address, OS::CommitPageSize());
180 |   return true;
181 | }
182 | 
183 | 
184 | void* VirtualMemory::ReserveRegion(size_t size) {
185 |   void* result = mmap(OS::GetRandomMmapAddr(),
186 |                       size,
187 |                       PROT_NONE,
188 |                       MAP_PRIVATE | MAP_ANONYMOUS | MAP_NORESERVE,
189 |                       kMmapFd,
190 |                       kMmapFdOffset);
191 | 
192 |   if (result == MAP_FAILED) return NULL;
193 | 
194 |   return result;
195 | }
196 | 
197 | 
198 | bool VirtualMemory::CommitRegion(void* base, size_t size, bool is_executable) {
199 |   int prot = PROT_READ | PROT_WRITE | (is_executable ? PROT_EXEC : 0);
200 |   if (MAP_FAILED == mmap(base,
201 |                          size,
202 |                          prot,
203 |                          MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED,
204 |                          kMmapFd,
205 |                          kMmapFdOffset)) {
206 |     return false;
207 |   }
208 | 
209 |   //UpdateAllocatedSpaceLimits(base, size);
210 |   return true;
211 | }
212 | 
213 | 
214 | bool VirtualMemory::UncommitRegion(void* base, size_t size) {
215 |   return mmap(base,
216 |               size,
217 |               PROT_NONE,
218 |               MAP_PRIVATE | MAP_ANONYMOUS | MAP_NORESERVE | MAP_FIXED,
219 |               kMmapFd,
220 |               kMmapFdOffset) != MAP_FAILED;
221 | }
222 | 
223 | 
224 | bool VirtualMemory::ReleaseRegion(void* base, size_t size) {
225 |   return munmap(base, size) == 0;
226 | }
227 | 
228 | 
229 | } }  // namespace rejit::internal
230 | 


--------------------------------------------------------------------------------
/src/platform/platform-macos.cc:
--------------------------------------------------------------------------------
  1 | // Copyright 2012 the V8 project authors. All rights reserved.
  2 | // Redistribution and use in source and binary forms, with or without
  3 | // modification, are permitted provided that the following conditions are
  4 | // met:
  5 | //
  6 | //     * Redistributions of source code must retain the above copyright
  7 | //       notice, this list of conditions and the following disclaimer.
  8 | //     * Redistributions in binary form must reproduce the above
  9 | //       copyright notice, this list of conditions and the following
 10 | //       disclaimer in the documentation and/or other materials provided
 11 | //       with the distribution.
 12 | //     * Neither the name of Google Inc. nor the names of its
 13 | //       contributors may be used to endorse or promote products derived
 14 | //       from this software without specific prior written permission.
 15 | //
 16 | // THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
 17 | // "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
 18 | // LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
 19 | // A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
 20 | // OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
 21 | // SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
 22 | // LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
 23 | // DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
 24 | // THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
 25 | // (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
 26 | // OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 27 | 
 28 | // Platform specific code for MacOS goes here. For the POSIX comaptible parts
 29 | // the implementation is in platform-posix.cc.
 30 | 
 31 | #include <dlfcn.h>
 32 | #include <unistd.h>
 33 | #include <sys/mman.h>
 34 | #include <mach/mach_init.h>
 35 | #include <mach-o/dyld.h>
 36 | #include <mach-o/getsect.h>
 37 | 
 38 | #include <AvailabilityMacros.h>
 39 | 
 40 | #include <pthread.h>
 41 | #include <semaphore.h>
 42 | #include <signal.h>
 43 | #include <libkern/OSAtomic.h>
 44 | #include <mach/mach.h>
 45 | #include <mach/semaphore.h>
 46 | #include <mach/task.h>
 47 | #include <mach/vm_statistics.h>
 48 | #include <sys/time.h>
 49 | #include <sys/resource.h>
 50 | #include <sys/types.h>
 51 | #include <sys/sysctl.h>
 52 | #include <stdarg.h>
 53 | #include <stdlib.h>
 54 | #include <string.h>
 55 | #include <errno.h>
 56 | 
 57 | #include "globals.h"
 58 | #include "utils.h"
 59 | #include "cpu.h"
 60 | 
 61 | #include <iostream>
 62 | 
 63 | #include "platform-posix.h"
 64 | #include "platform.h"
 65 | 
 66 | namespace rejit {
 67 | namespace internal {
 68 | 
 69 | intptr_t OS::CommitPageSize() {
 70 |   static intptr_t page_size = getpagesize();
 71 |   return page_size;
 72 | }
 73 | 
 74 | 
 75 | // Get rid of writable permission on code allocations.
 76 | void OS::ProtectCode(void* address, const size_t size) {
 77 |   mprotect(address, size, PROT_READ | PROT_EXEC);
 78 | }
 79 | 
 80 | 
 81 | // Create guard pages.
 82 | void OS::Guard(void* address, const size_t size) {
 83 |   mprotect(address, size, PROT_NONE);
 84 | }
 85 | void* OS::GetRandomMmapAddr() {
 86 |   // TODO(rames): Implement this.
 87 |   // UNIMPLEMENTED();
 88 |   return NULL;
 89 | }
 90 | 
 91 | size_t OS::AllocateAlignment() {
 92 |   return getpagesize();
 93 | }
 94 | 
 95 | 
 96 | void OS::Free(void* address, const size_t size) {
 97 |   // TODO(1240712): munmap has a return value which is ignored here.
 98 |   int result = munmap(address, size);
 99 |   USE(result);
100 |   ASSERT(result == 0);
101 | }
102 | 
103 | 
104 | uint64_t OS::CpuFeaturesImpliedByPlatform() {
105 |   // MacOSX requires all these to install so we can assume they are present.
106 |   // These constants are defined by the CPUid instructions.
107 |   const uint64_t one = 1;
108 |   return (one << SSE2) | (one << CMOV) | (one << RDTSC) | (one << CPUID);
109 | }
110 | 
111 | 
112 | static const int kMmapFd = VM_MAKE_TAG(255);
113 | static const int kMmapFdOffset = 0;
114 | 
115 | VirtualMemory::VirtualMemory() : address_(NULL), size_(0) { }
116 | 
117 | 
118 | VirtualMemory::VirtualMemory(size_t size)
119 |     : address_(ReserveRegion(size)), size_(size) { }
120 | 
121 | 
122 | VirtualMemory::VirtualMemory(size_t size, size_t alignment)
123 |     : address_(NULL), size_(0) {
124 |   ASSERT(IsAligned(alignment, static_cast<intptr_t>(OS::AllocateAlignment())));
125 |   size_t request_size = RoundUp(size + alignment,
126 |                                 static_cast<intptr_t>(OS::AllocateAlignment()));
127 |   void* reservation = mmap(OS::GetRandomMmapAddr(),
128 |                            request_size,
129 |                            PROT_NONE,
130 |                            MAP_PRIVATE | MAP_ANON | MAP_NORESERVE,
131 |                            kMmapFd,
132 |                            kMmapFdOffset);
133 |   if (reservation == MAP_FAILED) return;
134 | 
135 |   Address base = static_cast<Address>(reservation);
136 |   Address aligned_base = RoundUp(base, alignment);
137 |   ASSERT(base <= aligned_base);
138 | 
139 |   // Unmap extra memory reserved before and after the desired block.
140 |   if (aligned_base != base) {
141 |     size_t prefix_size = static_cast<size_t>(aligned_base - base);
142 |     OS::Free(base, prefix_size);
143 |     request_size -= prefix_size;
144 |   }
145 | 
146 |   size_t aligned_size = RoundUp(size, OS::AllocateAlignment());
147 |   ASSERT(aligned_size <= request_size);
148 | 
149 |   if (aligned_size != request_size) {
150 |     size_t suffix_size = request_size - aligned_size;
151 |     OS::Free(aligned_base + aligned_size, suffix_size);
152 |     request_size -= suffix_size;
153 |   }
154 | 
155 |   ASSERT(aligned_size == request_size);
156 | 
157 |   address_ = static_cast<void*>(aligned_base);
158 |   size_ = aligned_size;
159 | }
160 | 
161 | 
162 | VirtualMemory::~VirtualMemory() {
163 |   if (IsReserved()) {
164 |     bool result = ReleaseRegion(address(), size());
165 |     ASSERT(result);
166 |     USE(result);
167 |   }
168 | }
169 | 
170 | 
171 | void VirtualMemory::Reset() {
172 |   address_ = NULL;
173 |   size_ = 0;
174 | }
175 | 
176 | 
177 | void* VirtualMemory::ReserveRegion(size_t size) {
178 |   void* result = mmap(OS::GetRandomMmapAddr(),
179 |                       size,
180 |                       PROT_NONE,
181 |                       MAP_PRIVATE | MAP_ANON | MAP_NORESERVE,
182 |                       kMmapFd,
183 |                       kMmapFdOffset);
184 | 
185 |   if (result == MAP_FAILED) {
186 |     std::cout << "Failed mmap\n" << std::endl;
187 |   }
188 |   if (result == MAP_FAILED) return NULL;
189 | 
190 |   return result;
191 | }
192 | 
193 | 
194 | bool VirtualMemory::IsReserved() {
195 |   return address_ != NULL;
196 | }
197 | 
198 | 
199 | bool VirtualMemory::Commit(void* address, size_t size, bool is_executable) {
200 |   return CommitRegion(address, size, is_executable);
201 | }
202 | 
203 | 
204 | bool VirtualMemory::Guard(void* address) {
205 |   OS::Guard(address, OS::CommitPageSize());
206 |   return true;
207 | }
208 | 
209 | 
210 | bool VirtualMemory::CommitRegion(void* address,
211 |                                  size_t size,
212 |                                  bool is_executable) {
213 |   int prot = PROT_READ | PROT_WRITE | (is_executable ? PROT_EXEC : 0);
214 |   if (MAP_FAILED == mmap(address,
215 |                          size,
216 |                          prot,
217 |                          MAP_PRIVATE | MAP_ANON | MAP_FIXED,
218 |                          kMmapFd,
219 |                          kMmapFdOffset)) {
220 |     return false;
221 |   }
222 | 
223 |   // UpdateAllocatedSpaceLimits(address, size);
224 |   return true;
225 | }
226 | 
227 | 
228 | bool VirtualMemory::Uncommit(void* address, size_t size) {
229 |   return UncommitRegion(address, size);
230 | }
231 | 
232 | 
233 | bool VirtualMemory::UncommitRegion(void* address, size_t size) {
234 |   return mmap(address,
235 |               size,
236 |               PROT_NONE,
237 |               MAP_PRIVATE | MAP_ANON | MAP_NORESERVE | MAP_FIXED,
238 |               kMmapFd,
239 |               kMmapFdOffset) != MAP_FAILED;
240 | }
241 | 
242 | 
243 | bool VirtualMemory::ReleaseRegion(void* address, size_t size) {
244 |   return munmap(address, size) == 0;
245 | }
246 | 
247 | 
248 | } }  // namespace rejit::internal
249 | 


--------------------------------------------------------------------------------
/src/platform/platform-posix.cc:
--------------------------------------------------------------------------------
 1 | // Copyright 2012 the V8 project authors. All rights reserved.
 2 | // Redistribution and use in source and binary forms, with or without
 3 | // modification, are permitted provided that the following conditions are
 4 | // met:
 5 | //
 6 | //     * Redistributions of source code must retain the above copyright
 7 | //       notice, this list of conditions and the following disclaimer.
 8 | //     * Redistributions in binary form must reproduce the above
 9 | //       copyright notice, this list of conditions and the following
10 | //       disclaimer in the documentation and/or other materials provided
11 | //       with the distribution.
12 | //     * Neither the name of Google Inc. nor the names of its
13 | //       contributors may be used to endorse or promote products derived
14 | //       from this software without specific prior written permission.
15 | //
16 | // THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
17 | // "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
18 | // LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
19 | // A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
20 | // OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
21 | // SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
22 | // LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
23 | // DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
24 | // THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
25 | // (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
26 | // OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
27 | 
28 | // Platform specific code for POSIX goes here. This is not a platform on its
29 | // own but contains the parts which are the same across POSIX platforms Linux,
30 | // Mac OS, FreeBSD and OpenBSD.
31 | 
32 | #include "latform-posix.h"
33 | 
34 | #include <unistd.h>
35 | #include <errno.h>
36 | #include <time.h>
37 | 
38 | #include <sys/mman.h>
39 | #include <sys/socket.h>
40 | #include <sys/resource.h>
41 | #include <sys/time.h>
42 | #include <sys/types.h>
43 | #include <sys/stat.h>
44 | 
45 | #include <arpa/inet.h>
46 | #include <netinet/in.h>
47 | #include <netdb.h>
48 | 
49 | #include "platform.h"
50 | 
51 | namespace rejit {
52 | namespace internal {
53 | 
54 | intptr_t OS::CommitPageSize() {
55 |   static intptr_t page_size = getpagesize();
56 |   return page_size;
57 | }
58 | 
59 | 
60 | // Get rid of writable permission on code allocations.
61 | void OS::ProtectCode(void* address, const size_t size) {
62 |   mprotect(address, size, PROT_READ | PROT_EXEC);
63 | }
64 | 
65 | 
66 | // Create guard pages.
67 | void OS::Guard(void* address, const size_t size) {
68 |   mprotect(address, size, PROT_NONE);
69 | }
70 | void* OS::GetRandomMmapAddr() {
71 |   // TODO(rames): Implement this.
72 |   // UNIMPLEMENTED();
73 |   return NULL;
74 | }
75 | 
76 | 
77 | 
78 | } }  // namespace rejit::internal
79 | 


--------------------------------------------------------------------------------
/src/platform/platform-posix.h:
--------------------------------------------------------------------------------
 1 | // Copyright 2012 the V8 project authors. All rights reserved.
 2 | // Redistribution and use in source and binary forms, with or without
 3 | // modification, are permitted provided that the following conditions are
 4 | // met:
 5 | //
 6 | //     * Redistributions of source code must retain the above copyright
 7 | //       notice, this list of conditions and the following disclaimer.
 8 | //     * Redistributions in binary form must reproduce the above
 9 | //       copyright notice, this list of conditions and the following
10 | //       disclaimer in the documentation and/or other materials provided
11 | //       with the distribution.
12 | //     * Neither the name of Google Inc. nor the names of its
13 | //       contributors may be used to endorse or promote products derived
14 | //       from this software without specific prior written permission.
15 | //
16 | // THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
17 | // "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
18 | // LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
19 | // A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
20 | // OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
21 | // SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
22 | // LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
23 | // DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
24 | // THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
25 | // (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
26 | // OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
27 | 
28 | #ifndef V8_PLATFORM_POSIX_H_
29 | #define V8_PLATFORM_POSIX_H_
30 | 
31 | namespace v8 {
32 | namespace internal {
33 | 
34 | // Used by platform implementation files during OS::PostSetUp().
35 | void POSIXPostSetUp();
36 | 
37 | } }  // namespace v8::internal
38 | 
39 | #endif  // V8_PLATFORM_POSIX_H_
40 | 


--------------------------------------------------------------------------------
/src/rejit.cc:
--------------------------------------------------------------------------------
  1 | // Copyright (C) 2013 Alexandre Rames <alexandre@coreperf.com>
  2 | // rejit is free software: you can redistribute it and/or modify
  3 | // it under the terms of the GNU General Public License as published by
  4 | // the Free Software Foundation, either version 3 of the License, or
  5 | // (at your option) any later version.
  6 | //
  7 | // This program is distributed in the hope that it will be useful,
  8 | // but WITHOUT ANY WARRANTY; without even the implied warranty of
  9 | // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 10 | // GNU General Public License for more details.
 11 | //
 12 | // You should have received a copy of the GNU General Public License
 13 | // along with this program.  If not, see <http://www.gnu.org/licenses/>.
 14 | 
 15 | #include <iostream>
 16 | 
 17 | #include "rejit.h"
 18 | #include "checks.h"
 19 | #include "parser.h"
 20 | #include "codegen.h"
 21 | 
 22 | #include "macro-assembler.h"
 23 | 
 24 | using namespace rejit::internal;  // NOLINT
 25 | using namespace std;              // NOLINT
 26 | 
 27 | namespace rejit {
 28 | 
 29 | #define __ masm->
 30 | 
 31 | 
 32 | bool MatchFull(const char* regexp, const string& text) {
 33 |   return MatchFull(regexp, text.c_str(), text.size());
 34 | }
 35 | 
 36 | 
 37 | bool MatchFull(const char* regexp, const char* text, size_t text_size) {
 38 |   Regej re(regexp);
 39 |   return re.MatchFull(text, text_size);
 40 | }
 41 | 
 42 | 
 43 | bool MatchAnywhere(const char* regexp, const string& text) {
 44 |   return MatchAnywhere(regexp, text.c_str(), text.size());
 45 | }
 46 | 
 47 | 
 48 | bool MatchAnywhere(const char* regexp, const char* text, size_t text_size) {
 49 |   Regej re(regexp);
 50 |   return re.MatchAnywhere(text, text_size);
 51 | }
 52 | 
 53 | 
 54 | bool MatchFirst(const char* regexp, const string& text, Match* match) {
 55 |   return MatchFirst(regexp, text.c_str(), text.size(), match);
 56 | }
 57 | 
 58 | 
 59 | bool MatchFirst(const char* regexp, const char* text, size_t text_size,
 60 |                 Match* match) {
 61 |   Regej re(regexp);
 62 |   return re.MatchFirst(text, text_size, match);
 63 | }
 64 | 
 65 | 
 66 | size_t MatchAll(const char* regexp, const string& text,
 67 |                 std::vector<struct Match>* matches) {
 68 |   return MatchAll(regexp, text.c_str(), text.size(), matches);
 69 | }
 70 | 
 71 | 
 72 | size_t MatchAll(const char* regexp, const char* text, size_t text_size,
 73 |                 std::vector<struct Match>* matches) {
 74 |   Regej re(regexp);
 75 |   return re.MatchAll(text, text_size, matches);
 76 | }
 77 | 
 78 | 
 79 | size_t MatchAllCount(const char* regexp, const string& text) {
 80 |   return MatchAllCount(regexp, text.c_str(), text.size());
 81 | }
 82 | 
 83 | 
 84 | size_t MatchAllCount(const char* regexp, const char* text, size_t text_size) {
 85 |   Regej re(regexp);
 86 |   return re.MatchAllCount(text, text_size);
 87 | }
 88 | 
 89 | 
 90 | void Replace(Match to_replace, string& text, const string& with) {
 91 |   vector<Match> matches;
 92 |   matches.push_back(to_replace);
 93 |   return Replace(&matches, text, with);
 94 | }
 95 | 
 96 | 
 97 | void Replace(vector<Match>* to_replace, string& text, const string& with) {
 98 |   vector<Match>::iterator it;
 99 |   string res;
100 |   // The size of the original string is a good estimate for the size of the
101 |   // replaced string. To avoid memcpies when the size is growing we reserve
102 |   // slightly more.
103 |   res.reserve(text.size() + text.size() / 16);
104 |   const char* ptext = text.c_str();
105 |   for (it = to_replace->begin(); it < to_replace->end(); it++) {
106 |     res.append(ptext, (*it).begin - ptext);
107 |     ptext = (*it).end;
108 |     res.append(with);
109 |   }
110 |   res.append(ptext, text.c_str() + text.size() - ptext);
111 |   text.assign(res);
112 | }
113 | 
114 | 
115 | bool ReplaceFirst(const char* regexp, string& text, const string& with) {
116 |   Regej re(regexp);
117 |   return re.ReplaceFirst(text, with);
118 | }
119 | 
120 | 
121 | size_t ReplaceAll(const char* regexp, string& text, const string& with) {
122 |   Regej re(regexp);
123 |   return re.ReplaceAll(text, with);
124 | }
125 | 
126 | 
127 | Regej::Regej(const char* regexp) : regexp_(regexp), rinfo_(new RegexpInfo()) {
128 |   Parser parser;
129 |   status_ = parser.Parse(ERE, rinfo_, regexp_);
130 | }
131 | 
132 | 
133 | Regej::Regej(const string& regexp) :
134 |   regexp_(regexp.c_str()), rinfo_(new RegexpInfo()) {
135 |   Parser parser;
136 |   status_ = parser.Parse(ERE, rinfo_, regexp_);
137 | }
138 | 
139 | 
140 | Regej::~Regej() {
141 |   delete rinfo_;
142 | }
143 | 
144 | 
145 | bool Regej::MatchFull(const string& text) {
146 |   return MatchFull(text.c_str(), text.size());
147 | }
148 | 
149 | 
150 | bool Regej::MatchFull(const char* text, size_t text_size) {
151 |   if (!rinfo_->match_full_) {
152 |     if (!Compile(kMatchFull)) return false;
153 |   }
154 |   return rinfo_->match_full_(text, text_size);
155 | }
156 | 
157 | 
158 | bool Regej::MatchAnywhere(const string& text) {
159 |   return MatchAnywhere(text.c_str(), text.size());
160 | }
161 | 
162 | 
163 | bool Regej::MatchAnywhere(const char* text, size_t text_size) {
164 |   if (!rinfo_->match_anywhere_) {
165 |     if (!Compile(kMatchAnywhere)) return false;
166 |   }
167 |   return rinfo_->match_anywhere_(text, text_size);
168 | }
169 | 
170 | 
171 | bool Regej::MatchFirst(const string& text, Match* match) {
172 |   return MatchFirst(text.c_str(), text.size(), match);
173 | }
174 | 
175 | 
176 | bool Regej::MatchFirst(const char* text, size_t text_size, Match* match) {
177 |   if (!rinfo_->match_first_) {
178 |     if (!Compile(kMatchFirst)) return false;
179 |   }
180 |   return rinfo_->match_first_(text, text_size, match);
181 | }
182 | 
183 | 
184 | size_t Regej::MatchAll(const string& text, vector<Match>* matches) {
185 |   return MatchAll(text.c_str(), text.size(), matches);
186 | }
187 | 
188 | 
189 | size_t Regej::MatchAll(const char* text, size_t text_size, vector<Match>* matches) {
190 |   if (!rinfo_->match_all_) {
191 |     if (!Compile(kMatchAll)) return 0;
192 |   }
193 |   rinfo_->match_all_(text, text_size, matches);
194 |   return matches->size();
195 | }
196 | 
197 | 
198 | size_t Regej::MatchAllCount(const string& text) {
199 |   return MatchAllCount(text.c_str(), text.size());
200 | }
201 | 
202 | 
203 | size_t Regej::MatchAllCount(const char* text, size_t text_size) {
204 |   // TODO: Test if using a separate function not registering matches in a vector
205 |   // is faster.
206 |   vector<Match> matches;
207 |   return MatchAll(text, text_size, &matches);
208 | }
209 | 
210 | 
211 | bool Regej::ReplaceFirst(string& text, const string& with) {
212 |   Match match;
213 |   bool match_found = MatchFirst(text, &match);
214 |   if (match_found) {
215 |     Replace(match, text, with);
216 |   }
217 |   return match_found;
218 | }
219 | 
220 | 
221 | size_t Regej::ReplaceAll(string& text, const string& with) {
222 |   vector<Match> matches;
223 |   MatchAll(text, &matches);
224 |   Replace(&matches, text, with);
225 |   return matches.size();
226 | }
227 | 
228 | 
229 | bool Regej::Compile(MatchType match_type) {
230 |   if (status() != RejitSuccess) {
231 |     return false;
232 |   }
233 | 
234 |   Codegen codegen;
235 |   VirtualMemory* vmem = codegen.Compile(rinfo_, match_type);
236 | 
237 |   switch (match_type) {
238 |     case kMatchFull:
239 |       rinfo_->vmem_match_full_ = vmem;
240 |       rinfo_->match_full_ =
241 |         FUNCTION_CAST<MatchFullFunc>(Address(vmem->address()));
242 |       break;
243 | 
244 |     case kMatchAnywhere:
245 |       rinfo_->vmem_match_anywhere_ = vmem;
246 |       rinfo_->match_anywhere_ =
247 |         FUNCTION_CAST<MatchAnywhereFunc>(Address(vmem->address()));
248 |       break;
249 | 
250 |     case kMatchFirst:
251 |       rinfo_->vmem_match_first_ = vmem;
252 |       rinfo_->match_first_ =
253 |         FUNCTION_CAST<MatchFirstFunc>(Address(vmem->address()));
254 |       break;
255 | 
256 |     case kMatchAll:
257 |       rinfo_->vmem_match_all_ = vmem;
258 |       rinfo_->match_all_ =
259 |         FUNCTION_CAST<MatchAllFunc>(Address(vmem->address()));
260 |       break;
261 | 
262 |     default:
263 |       UNREACHABLE();
264 |   }
265 | 
266 |   return vmem != NULL;
267 | }
268 | 
269 | 
270 | }  // namespace rejit
271 | 


--------------------------------------------------------------------------------
/src/suffix_trees.cc:
--------------------------------------------------------------------------------
  1 | // Copyright (C) 2013 Alexandre Rames <alexandre@coreperf.com>
  2 | // rejit is free software: you can redistribute it and/or modify
  3 | // it under the terms of the GNU General Public License as published by
  4 | // the Free Software Foundation, either version 3 of the License, or
  5 | // (at your option) any later version.
  6 | //
  7 | // This program is distributed in the hope that it will be useful,
  8 | // but WITHOUT ANY WARRANTY; without even the implied warranty of
  9 | // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 10 | // GNU General Public License for more details.
 11 | //
 12 | // You should have received a copy of the GNU General Public License
 13 | // along with this program.  If not, see <http://www.gnu.org/licenses/>.
 14 | 
 15 | #include "suffix_trees.h"
 16 | 
 17 | #include "utils.h"
 18 | 
 19 | 
 20 | namespace rejit {
 21 | namespace internal {
 22 | 
 23 | 
 24 | SuffixTree::~SuffixTree() {
 25 |   for (pair<char, SuffixTree*> edge : suffixes_) {
 26 |     delete edge.second;
 27 |   }
 28 | }
 29 | 
 30 | ostream& SuffixTree::OutputToIOStream(ostream& stream) const {
 31 |   if (str_start_ >= 0 && str_end_ >= 0) {
 32 |     Indent(stream) << string(str_, str_start_, str_end_ - str_start_);
 33 |   }
 34 | 
 35 |   cout << "\t(" << suffixed_mcs_.size()<< ")";
 36 |   if (terminated_mcs_.size()) {
 37 |     cout << "\t(terminates mcs:";
 38 |     for (const MultipleChar *mc : terminated_mcs_) {
 39 |       cout << " \"" << *mc << "\"";
 40 |     }
 41 |     cout << ")";
 42 |   }
 43 |   cout << endl;
 44 | 
 45 |   {
 46 |     // Align the suffixes after the end of the string for this node.
 47 |     IndentScope is(str_end_ - str_start_);
 48 |     for (pair<char, SuffixTree*> edge : suffixes_) {
 49 |       cout << *(edge.second);
 50 |     }
 51 |   }
 52 |   return stream;
 53 | }
 54 | 
 55 | 
 56 | // This cannot be declared inline as the '<<' operator has not been overloaded
 57 | // before the declaration site.
 58 | void SuffixTree::print() { cout << *this; }
 59 | 
 60 | 
 61 | const SuffixTree *lowest_common_ancestor(SuffixTree *root,
 62 |                                          unsigned n_mcs) {
 63 |   const SuffixTree *n = NULL, *lowest = NULL;
 64 |   for (auto edge : *root->suffixes()) {
 65 |     n = lowest_common_ancestor(edge.second, n_mcs);
 66 |     if (n && (!lowest || n->active_length() > lowest->active_length())) {
 67 |       lowest = n;
 68 |     }
 69 |     root->suffixed_mcs()->insert(edge.second->suffixed_mcs()->begin(),
 70 |                                  edge.second->suffixed_mcs()->end());
 71 |   }
 72 |   if (lowest) {
 73 |     return lowest;
 74 |   }
 75 |   for (auto mc : *root->terminated_mcs()) {
 76 |     root->suffixed_mcs()->insert(mc);
 77 |   }
 78 |   if (root->suffixed_mcs()->size() == n_mcs) {
 79 |     lowest = root;
 80 |   }
 81 |   return lowest;
 82 | }
 83 | 
 84 | 
 85 | // TODO: Is there no standard library function for this?!
 86 | static size_t str_diff_index(const char *s1, const char *s2) {
 87 |   size_t i = 0;
 88 |   while (s1[i] && s2[i] && s1[i] == s2[i]) {
 89 |     i++;
 90 |   }
 91 |   return i;
 92 | }
 93 | 
 94 | 
 95 | int SuffixTreeBuilder::walk_down(const char *str) {
 96 |   size_t str_len = strlen(str);
 97 |   map<char,SuffixTree*>::iterator edge_it;
 98 |   SuffixTree *edge_node;
 99 |   string edge_str;
100 |   size_t diff_index = 0;
101 | 
102 |   while (active_length_ < str_len &&
103 |          active_node_->suffixes_.end() !=
104 |          (edge_it = active_node_->suffixes_.find(str[active_length_]))) {
105 |     edge_node = (*edge_it).second;
106 |     edge_str = edge_node->edge_string();
107 |     diff_index = str_diff_index(edge_str.c_str(), str + active_length_);
108 | 
109 |     if (diff_index == edge_str.length()) {
110 |       // We consumed the whole edge to go to the next node.
111 |       active_node_ = (*edge_it).second;
112 |       active_length_ += diff_index;
113 |       diff_index = 0;
114 |       continue;
115 |     } else {
116 |       // Stay on the current active node. 'diff_index' indicates at what point a
117 |       // split is necessary.
118 |       return diff_index;
119 |     }
120 |   }
121 | 
122 |   return diff_index;
123 | }
124 | 
125 | 
126 | SuffixTree *SuffixTreeBuilder::append_mc(const MultipleChar *mc) {
127 |   string *str_ = new string(mc->chars(), mc->chars_length());
128 |   allocated_strings_.push_back(str_);
129 |   const char *str = str_->c_str();
130 |   size_t str_len = str_->length();
131 |   size_t offset;
132 | 
133 |   for (size_t i = 0; i < str_len; i++) {
134 | #ifdef TRACE_SUFFIX_TREES_CONSTRUCTION
135 |     cout << "--------------------" << endl;
136 |     cout << "inserting " << str + i << endl << endl;;
137 | #endif
138 | 
139 |     // TODO: We currently only insert from the root. Usage of suffix pointers
140 |     // will be added in a later commit.
141 |     active_node_ = root_;
142 |     active_length_ = 0;
143 | 
144 |     offset = walk_down(str + i);
145 | 
146 |     if (offset) {
147 | 
148 |       // We need to split along the current edge.
149 |       map<char,SuffixTree*>::iterator active_edge_it =
150 |         active_node_->suffixes_.find(str[i + active_length_]);
151 |       SuffixTree *edge_node = (*active_edge_it).second;
152 | 
153 |       // Split the existing edge in two.
154 |       SuffixTree *middle_node =
155 |         new SuffixTree(str, i + active_length_, i + active_length_ + offset,
156 |                        active_length_);
157 |       active_node_->suffixes_[str[i + active_length_]] = middle_node;
158 |       edge_node->str_start_ = edge_node->str_start_ + offset;
159 |       middle_node->suffixes_[edge_node->first_edge_char()] = edge_node;
160 | 
161 |       if (i + active_length_ + offset < str_len) {
162 |         // Create a new node for the remaining characters in the string.
163 |         SuffixTree *new_node =
164 |           new SuffixTree(str, i + active_length_ + offset, str_len,
165 |                          active_length_);
166 |         middle_node->suffixes_[str[i + active_length_ + offset]] = new_node;
167 |         new_node->terminated_mcs_.insert(mc);
168 |       } else {
169 |         // The string ends on this middle node.
170 |         middle_node->terminated_mcs_.insert(mc);
171 |       }
172 | 
173 |     } else {
174 |       if (i + active_length_ == str_len) {
175 |         // No more characters to add.
176 |         active_node_->terminated_mcs_.insert(mc);
177 |       } else {
178 |         SuffixTree *new_node =
179 |           new SuffixTree(str, i + active_length_, str_len, active_length_);
180 |         active_node_->suffixes_[str[i + active_length_]] = new_node;
181 |         new_node->terminated_mcs_.insert(mc);
182 |       }
183 |     }
184 | 
185 | #ifdef TRACE_SUFFIX_TREES_CONSTRUCTION
186 |     cout << *root_;
187 |     cout << "--------------------" << endl;
188 | #endif
189 |   }
190 | 
191 |   return root_;
192 | }
193 | 
194 | 
195 | } }  // namespace rejit::internal
196 | 


--------------------------------------------------------------------------------
/src/suffix_trees.h:
--------------------------------------------------------------------------------
  1 | // Copyright (C) 2013 Alexandre Rames <alexandre@coreperf.com>
  2 | // rejit is free software: you can redistribute it and/or modify
  3 | // it under the terms of the GNU General Public License as published by
  4 | // the Free Software Foundation, either version 3 of the License, or
  5 | // (at your option) any later version.
  6 | //
  7 | // This program is distributed in the hope that it will be useful,
  8 | // but WITHOUT ANY WARRANTY; without even the implied warranty of
  9 | // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 10 | // GNU General Public License for more details.
 11 | //
 12 | // You should have received a copy of the GNU General Public License
 13 | // along with this program.  If not, see <http://www.gnu.org/licenses/>.
 14 | 
 15 | 
 16 | // Simple suffix tree implementation. This will be used to try to simplify
 17 | // alternations of strings in fast-forward elements.
 18 | // TODO: The current implementation always insert starting from the root.
 19 | // Improve the implementation, maybe using suffix pointers (Ukkonen's
 20 | // algorithm).
 21 | 
 22 | #ifndef REJIT_SUFFIX_TREES_H_
 23 | #define REJIT_SUFFIX_TREES_H_
 24 | 
 25 | #include <map>
 26 | #include <set>
 27 | #include <string>
 28 | 
 29 | #include "regexp.h"
 30 | 
 31 | using namespace std;
 32 | 
 33 | namespace rejit {
 34 | namespace internal {
 35 | 
 36 | 
 37 | class SuffixTree {
 38 |  public:
 39 |   explicit SuffixTree() :
 40 |       str_(NULL), str_start_(-1), str_end_(-1), active_length_(0) {}
 41 |   // `parent_active_length` is the active length of the parent node to which
 42 |   // this node is being appended.
 43 |   explicit SuffixTree(const char *str, int start, int end,
 44 |                       int parent_active_length = 0) :
 45 |       str_(str), str_start_(start), str_end_(end),
 46 |       active_length_(parent_active_length + end - start) {}
 47 |   virtual ~SuffixTree();
 48 | 
 49 |   inline string edge_string() {
 50 |     return string(str_, str_start_, str_end_ - str_start_);
 51 |   }
 52 |   inline char first_edge_char() { return str_[str_start_]; }
 53 | 
 54 |   virtual ostream& OutputToIOStream(ostream& stream) const;
 55 |   void print();
 56 | 
 57 |   // Accessors.
 58 |   const char *str() const { return str_; }
 59 |   const int str_start() const { return str_start_; }
 60 |   const int str_end() const { return str_end_; }
 61 | 
 62 |   inline const map<char,SuffixTree*> *suffixes() const { return &suffixes_; }
 63 |   inline const set<const MultipleChar*> *terminated_mcs() const {
 64 |     return &terminated_mcs_;
 65 |   }
 66 |   inline set<const MultipleChar*> *suffixed_mcs() { return &suffixed_mcs_; }
 67 |   inline const int active_length() const { return active_length_; }
 68 | 
 69 |  private:
 70 |   // Edges leading to suffix nodes.
 71 |   map<char, SuffixTree*> suffixes_;
 72 |   // Characters for the edge leading to this node.
 73 |   const char *str_;
 74 |   int str_start_;
 75 |   int str_end_;
 76 |   // The set of strings fow which this node is a suffix.
 77 |   set<const MultipleChar*> terminated_mcs_;
 78 |   set<const MultipleChar*> suffixed_mcs_;
 79 | 
 80 |   unsigned active_length_;
 81 | 
 82 |   friend class SuffixTreeBuilder;
 83 | };
 84 | 
 85 | 
 86 | // Find the lowest common ancestors for mcs in this tree.
 87 | // `n_mcs` is passed explicitly to avoid depending on the builder.
 88 | const SuffixTree *lowest_common_ancestor(SuffixTree *root,
 89 |                                          unsigned n_mcs);
 90 | 
 91 | 
 92 | // Overload the '<<' operator to allow convenient printing to an iostream.
 93 | inline ostream& operator<<(ostream& stream, const SuffixTree& suffix_tree) {
 94 |   suffix_tree.OutputToIOStream(stream);
 95 |   return stream;
 96 | }
 97 | 
 98 | 
 99 | class SuffixTreeBuilder {
100 |  public:
101 |   SuffixTreeBuilder() :
102 |       root_(new SuffixTree()), active_length_(0) {
103 |     active_node_ = root_;
104 |   }
105 |   ~SuffixTreeBuilder() {
106 |     delete root_;
107 |     for (string *s : allocated_strings_) {
108 |       delete s;
109 |     }
110 |   }
111 | 
112 |   // From the active node, walk toward the node at which we must perform the
113 |   // next suffix insertion (which will become the new active node).
114 |   // Returns the offset at which the insertion must be done on the current edge.
115 |   int walk_down(const char *str);
116 | 
117 |   // Build the suffixes for this string into the existing suffix tree.
118 |   SuffixTree *append_mc(const MultipleChar *mc);
119 | 
120 |   inline SuffixTree *root() const { return root_; }
121 | 
122 |  private:
123 |   SuffixTree *root_;
124 |   SuffixTree *active_node_;
125 |   unsigned active_length_;
126 | 
127 |   vector<string*> allocated_strings_;
128 | };
129 | 
130 | 
131 | } }  // namespace rejit::internal
132 | 
133 | #endif
134 | 


--------------------------------------------------------------------------------
/src/utils.cc:
--------------------------------------------------------------------------------
 1 | // Copyright (C) 2013 Alexandre Rames <alexandre@coreperf.com>
 2 | // rejit is free software: you can redistribute it and/or modify
 3 | // it under the terms of the GNU General Public License as published by
 4 | // the Free Software Foundation, either version 3 of the License, or
 5 | // (at your option) any later version.
 6 | //
 7 | // This program is distributed in the hope that it will be useful,
 8 | // but WITHOUT ANY WARRANTY; without even the implied warranty of
 9 | // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
10 | // GNU General Public License for more details.
11 | //
12 | // You should have received a copy of the GNU General Public License
13 | // along with this program.  If not, see <http://www.gnu.org/licenses/>.
14 | 
15 | #include <iostream>
16 | 
17 | #include "utils.h"
18 | #include "globals.h"
19 | #include "rejit.h"
20 | 
21 | namespace rejit {
22 | 
23 | char rejit_status_string_buf[STATUS_STRING_SIZE];
24 | char * const rejit_status_string = rejit_status_string_buf;
25 | 
26 | namespace internal {
27 | 
28 | int IndentationLevel = 0;
29 | 
30 | 
31 | uint64_t FirstBytesMask(int n) {
32 |   if (n >= kPointerSize) return kMaxUInt64;
33 |   uint64_t mask = (1ULL << (n * kBitsPerByte)) - 1ULL;
34 |   return mask;
35 | }
36 | 
37 | } }  // namespace rejit::internal
38 | 
39 | 


--------------------------------------------------------------------------------
/src/utils.h:
--------------------------------------------------------------------------------
  1 | // Copyright (C) 2013 Alexandre Rames <alexandre@coreperf.com>
  2 | // rejit is free software: you can redistribute it and/or modify
  3 | // it under the terms of the GNU General Public License as published by
  4 | // the Free Software Foundation, either version 3 of the License, or
  5 | // (at your option) any later version.
  6 | //
  7 | // This program is distributed in the hope that it will be useful,
  8 | // but WITHOUT ANY WARRANTY; without even the implied warranty of
  9 | // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 10 | // GNU General Public License for more details.
 11 | //
 12 | // You should have received a copy of the GNU General Public License
 13 | // along with this program.  If not, see <http://www.gnu.org/licenses/>.
 14 | 
 15 | #ifndef REJIT_UTILS_H
 16 | #define REJIT_UTILS_H
 17 | 
 18 | #include <stdint.h>
 19 | #include <string>
 20 | #include "globals.h"
 21 | 
 22 | using namespace std;  // NOLINT
 23 | 
 24 | namespace rejit {
 25 | namespace internal {
 26 | 
 27 | #define STATUS_STRING_SIZE 200
 28 | 
 29 | // Indentation helpers ---------------------------------------------------------
 30 | extern int IndentationLevel;
 31 | 
 32 | class IndentScope {
 33 |  public:
 34 |   IndentScope(int _indent = 1) : indent(_indent) {
 35 |     IndentationLevel = max(0, IndentationLevel + indent);
 36 |   }
 37 |   ~IndentScope() {
 38 |     IndentationLevel = max(0, IndentationLevel - indent);
 39 |   }
 40 |  private:
 41 |   int indent;
 42 | };
 43 | 
 44 | inline ostream& Indent(ostream& stream) {
 45 |   return stream << string(IndentationLevel, ' ');
 46 | }
 47 | 
 48 | 
 49 | // Addressing ------------------------------------------------------------------
 50 | 
 51 | template <typename T, typename U>
 52 | inline bool IsAligned(T value, U alignment) {
 53 |   return (value & (alignment - 1)) == 0;
 54 | }
 55 | 
 56 | // Compute the 0-relative offset of some absolute value x of type T.
 57 | // This allows conversion of Addresses and integral types into
 58 | // 0-relative int offsets.
 59 | template <typename T>
 60 | inline intptr_t OffsetFrom(T x) {
 61 |   return x - static_cast<T>(0);
 62 | }
 63 | 
 64 | // Compute the absolute value of type T for some 0-relative offset x.
 65 | // This allows conversion of 0-relative int offsets into Addresses and
 66 | // integral types.
 67 | template <typename T>
 68 | inline T AddressFrom(intptr_t x) {
 69 |   return static_cast<T>(static_cast<T>(0) + x);
 70 | }
 71 | 
 72 | // The type-based aliasing rule allows the compiler to assume that pointers of
 73 | // different types (for some definition of different) never alias each other.
 74 | // Thus the following code does not work:
 75 | //
 76 | // float f = foo();
 77 | // int fbits = *(int*)(&f);
 78 | //
 79 | // The compiler 'knows' that the int pointer can't refer to f since the types
 80 | // don't match, so the compiler may cache f in a register, leaving random data
 81 | // in fbits.  Using C++ style casts makes no difference, however a pointer to
 82 | // char data is assumed to alias any other pointer.  This is the 'memcpy
 83 | // exception'.
 84 | //
 85 | // Bit_cast uses the memcpy exception to move the bits from a variable of one
 86 | // type of a variable of another type.  Of course the end result is likely to
 87 | // be implementation dependent.  Most compilers (gcc-4.2 and MSVC 2005)
 88 | // will completely optimize BitCast away.
 89 | //
 90 | // There is an additional use for BitCast.
 91 | // Recent gccs will warn when they see casts that may result in breakage due to
 92 | // the type-based aliasing rule.  If you have checked that there is no breakage
 93 | // you can use BitCast to cast one pointer type to another.  This confuses gcc
 94 | // enough that it can no longer see that you have cast one pointer type to
 95 | // another thus avoiding the warning.
 96 | 
 97 | // We need different implementations of BitCast for pointer and non-pointer
 98 | // values. We use partial specialization of auxiliary struct to work around
 99 | // issues with template functions overloading.
100 | template <class Dest, class Source>
101 | struct BitCastHelper {
102 |   STATIC_ASSERT(sizeof(Dest) == sizeof(Source));
103 | 
104 |   inline static Dest cast(const Source& source) {
105 |     Dest dest;
106 |     memcpy(&dest, &source, sizeof(dest));
107 |     return dest;
108 |   }
109 | };
110 | 
111 | template <class Dest, class Source>
112 | struct BitCastHelper<Dest, Source*> {
113 |   inline static Dest cast(Source* source) {
114 |     return BitCastHelper<Dest, uintptr_t>::
115 |         cast(reinterpret_cast<uintptr_t>(source));
116 |   }
117 | };
118 | 
119 | template <class Dest, class Source>
120 | inline Dest BitCast(const Source& source);
121 | 
122 | template <class Dest, class Source>
123 | inline Dest BitCast(const Source& source) {
124 |   return BitCastHelper<Dest, Source>::cast(source);
125 | }
126 | 
127 | //#define offsetof(type, member) __builtin_offsetof(type, member)
128 | 
129 | 
130 | // Arithmetic ------------------------------------------------------------------
131 | // TODO(rames): Use good implementations for all arithmetic utils.
132 | 
133 | uint64_t FirstBytesMask(int n);
134 | inline uint64_t FirstCharsMask(int n) { return FirstBytesMask(n * kCharSize); }
135 | 
136 | // Returns true iff x a power of 2. Cannot be used with the
137 | // maximally negative value of the type T (the -1 overflows).
138 | template <typename T>
139 | inline bool IsPowerOf2(T x) {
140 |   return x && ((x & (x - 1)) == 0);
141 | }
142 | 
143 | inline unsigned WhichPowerOf2(uint64_t x) {
144 |   ASSERT(x && IsPowerOf2(x));
145 |   unsigned power = 0;
146 |   power += (x >= (1ULL << 32)) * 32;
147 |   power += (x >= (1ULL << 16)) * 16;
148 |   power += (x >= (1ULL <<  8)) *  8;
149 |   power += (x >= (1ULL <<  4)) *  4;
150 |   power += (x >= (1ULL <<  2)) *  2;
151 |   power += (x >= (1ULL <<  1)) *  1;
152 |   return power;
153 | }
154 | 
155 | inline unsigned NumberOfBitsSet(uint64_t x) {
156 |   return __builtin_popcountl(x);
157 | }
158 | 
159 | // Return the largest multiple of m which is <= x.
160 | template <typename T>
161 | inline T RoundDown(T x, intptr_t m) {
162 |   ASSERT(IsPowerOf2(m));
163 |   return AddressFrom<T>(OffsetFrom(x) & -m);
164 | }
165 | 
166 | // Return the smallest multiple of m which is >= x.
167 | template <typename T>
168 | inline T RoundUp(T x, intptr_t m) {
169 |   return RoundDown<T>(static_cast<T>(x + m - 1), m);
170 | }
171 | 
172 | #define REPEAT_1_T0_63(M) \
173 |   M(1) M(2) M(3) M(4) M(5) M(6) M(7) M(8) M(9) M(10) M(11) M(12) M(13) M(14) \
174 |   M(15) M(16) M(17) M(18) M(19) M(20) M(21) M(22) M(23) M(24) M(25) M(26)    \
175 |   M(27) M(28) M(29) M(30) M(31) M(32) M(33) M(34) M(35) M(36) M(37) M(38)    \
176 |   M(39) M(40) M(41) M(42) M(43) M(44) M(45) M(46) M(47) M(48) M(49) M(50)    \
177 |   M(51) M(52) M(53) M(54) M(55) M(56) M(57) M(58) M(59) M(60) M(61) M(62)    \
178 |   M(63)
179 | 
180 | inline bool is_intn(int64_t x, unsigned n) {
181 |   return -(1LL << (n-1)) <= x && x < (1LL << (n-1));
182 | }
183 | 
184 | #define DEFINE_IS_INTN(n) \
185 |   inline bool is_int##n(int64_t x) { return is_intn(x, n); }
186 | REPEAT_1_T0_63(DEFINE_IS_INTN)
187 | #undef DEFINE_IS_INTN
188 | 
189 | inline bool is_uintn(int64_t x, int n) {
190 |   return (x & -(1LL << n)) == 0;
191 | }
192 | 
193 | #define DEFINE_IS_UINTN(n) \
194 |   inline bool is_uint##n(int64_t x) { return is_uintn(x, n); }
195 | REPEAT_1_T0_63(DEFINE_IS_UINTN)
196 | #undef DEFINE_IS_UINTN
197 | 
198 | #undef REPEAT_1_T0_63
199 | 
200 | // Range iterators -------------------------------------------------------------
201 | template <typename Iterator>
202 | class iterator_pair {
203 |  public:
204 |   iterator_pair(Iterator begin, Iterator end) : begin_(begin), end_(end) {}
205 | 
206 |   Iterator begin() const { return begin_; }
207 |   Iterator end() const { return end_; }
208 | 
209 |  private:
210 |   Iterator begin_;
211 |   Iterator end_;
212 | };
213 | 
214 | template <typename Iterator>
215 | iterator_pair<Iterator> range(Iterator begin, Iterator end) {
216 |   return iterator_pair<Iterator>(begin, end);
217 | }
218 | 
219 | template <typename Iterator>
220 | iterator_pair<Iterator> range(pair<Iterator, Iterator> it_pair) {
221 |   return iterator_pair<Iterator>(it_pair.first, it_pair.second);
222 | }
223 | 
224 | } }  // namespace rejit::internal
225 | 
226 | #endif  // REJIT_UTILS_H
227 | 
228 | 


--------------------------------------------------------------------------------
/src/x64/assembler-x64-inl.h:
--------------------------------------------------------------------------------
  1 | // Imported from v8 git repository:
  2 | // commit 32777d04209b0ee6b21e12a8e7d7ace59bce3c23
  3 | // Date:   Thu Feb 14 15:12:49 2013 +0000
  4 | // svn id: 13673 ce2b1a6d-e550-0410-aec6-3dcde31c8c00
  5 | 
  6 | // Copyright 2012 the V8 project authors. All rights reserved.
  7 | // Redistribution and use in source and binary forms, with or without
  8 | // modification, are permitted provided that the following conditions are
  9 | // met:
 10 | //
 11 | //     * Redistributions of source code must retain the above copyright
 12 | //       notice, this list of conditions and the following disclaimer.
 13 | //     * Redistributions in binary form must reproduce the above
 14 | //       copyright notice, this list of conditions and the following
 15 | //       disclaimer in the documentation and/or other materials provided
 16 | //       with the distribution.
 17 | //     * Neither the name of Google Inc. nor the names of its
 18 | //       contributors may be used to endorse or promote products derived
 19 | //       from this software without specific prior written permission.
 20 | //
 21 | // THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
 22 | // "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
 23 | // LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
 24 | // A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
 25 | // OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
 26 | // SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
 27 | // LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
 28 | // DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
 29 | // THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
 30 | // (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
 31 | // OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 32 | 
 33 | #ifndef REJIT_X64_ASSEMBLER_X64_INL_H_
 34 | #define REJIT_X64_ASSEMBLER_X64_INL_H_
 35 | 
 36 | #include "x64/assembler-x64.h"
 37 | 
 38 | #include "cpu.h"
 39 | #include "memory.h"
 40 | 
 41 | namespace rejit {
 42 | namespace internal {
 43 | 
 44 | 
 45 | // -----------------------------------------------------------------------------
 46 | // Implementation of Assembler
 47 | 
 48 | 
 49 | void Assembler::emitq(uint64_t x) {
 50 |   Memory::uint64_at(pc_) = x;
 51 |   pc_ += sizeof(uint64_t);
 52 | }
 53 | 
 54 | 
 55 | void Assembler::emitl(uint32_t x) {
 56 |   Memory::uint32_at(pc_) = x;
 57 |   pc_ += sizeof(uint32_t);
 58 | }
 59 | 
 60 | 
 61 | void Assembler::emitw(uint16_t x) {
 62 |   Memory::uint16_at(pc_) = x;
 63 |   pc_ += sizeof(uint16_t);
 64 | }
 65 | 
 66 | 
 67 | void Assembler::emit_rex_64(Register reg, Register rm_reg) {
 68 |   emit(0x48 | reg.high_bit() << 2 | rm_reg.high_bit());
 69 | }
 70 | 
 71 | 
 72 | void Assembler::emit_rex_64(XMMRegister reg, Register rm_reg) {
 73 |   emit(0x48 | (reg.code() & 0x8) >> 1 | rm_reg.code() >> 3);
 74 | }
 75 | 
 76 | 
 77 | void Assembler::emit_rex_64(Register reg, XMMRegister rm_reg) {
 78 |   emit(0x48 | (reg.code() & 0x8) >> 1 | rm_reg.code() >> 3);
 79 | }
 80 | 
 81 | 
 82 | void Assembler::emit_rex_64(Register reg, const Operand& op) {
 83 |   emit(0x48 | reg.high_bit() << 2 | op.rex_);
 84 | }
 85 | 
 86 | 
 87 | void Assembler::emit_rex_64(XMMRegister reg, const Operand& op) {
 88 |   emit(0x48 | (reg.code() & 0x8) >> 1 | op.rex_);
 89 | }
 90 | 
 91 | 
 92 | void Assembler::emit_rex_64(Register rm_reg) {
 93 |   ASSERT((rm_reg.code() & 0xf) == rm_reg.code());
 94 |   emit(0x48 | rm_reg.high_bit());
 95 | }
 96 | 
 97 | 
 98 | void Assembler::emit_rex_64(const Operand& op) {
 99 |   emit(0x48 | op.rex_);
100 | }
101 | 
102 | 
103 | void Assembler::emit_rex_32(Register reg, Register rm_reg) {
104 |   emit(0x40 | reg.high_bit() << 2 | rm_reg.high_bit());
105 | }
106 | 
107 | 
108 | void Assembler::emit_rex_32(Register reg, const Operand& op) {
109 |   emit(0x40 | reg.high_bit() << 2  | op.rex_);
110 | }
111 | 
112 | 
113 | void Assembler::emit_rex_32(Register rm_reg) {
114 |   emit(0x40 | rm_reg.high_bit());
115 | }
116 | 
117 | 
118 | void Assembler::emit_rex_32(const Operand& op) {
119 |   emit(0x40 | op.rex_);
120 | }
121 | 
122 | 
123 | void Assembler::emit_optional_rex_32(Register reg, Register rm_reg) {
124 |   byte rex_bits = reg.high_bit() << 2 | rm_reg.high_bit();
125 |   if (rex_bits != 0) emit(0x40 | rex_bits);
126 | }
127 | 
128 | 
129 | void Assembler::emit_optional_rex_32(Register reg, const Operand& op) {
130 |   byte rex_bits =  reg.high_bit() << 2 | op.rex_;
131 |   if (rex_bits != 0) emit(0x40 | rex_bits);
132 | }
133 | 
134 | 
135 | void Assembler::emit_optional_rex_32(XMMRegister reg, const Operand& op) {
136 |   byte rex_bits =  (reg.code() & 0x8) >> 1 | op.rex_;
137 |   if (rex_bits != 0) emit(0x40 | rex_bits);
138 | }
139 | 
140 | 
141 | void Assembler::emit_optional_rex_32(XMMRegister reg, XMMRegister base) {
142 |   byte rex_bits =  (reg.code() & 0x8) >> 1 | (base.code() & 0x8) >> 3;
143 |   if (rex_bits != 0) emit(0x40 | rex_bits);
144 | }
145 | 
146 | 
147 | void Assembler::emit_optional_rex_32(XMMRegister reg, Register base) {
148 |   byte rex_bits =  (reg.code() & 0x8) >> 1 | (base.code() & 0x8) >> 3;
149 |   if (rex_bits != 0) emit(0x40 | rex_bits);
150 | }
151 | 
152 | 
153 | void Assembler::emit_optional_rex_32(Register reg, XMMRegister base) {
154 |   byte rex_bits =  (reg.code() & 0x8) >> 1 | (base.code() & 0x8) >> 3;
155 |   if (rex_bits != 0) emit(0x40 | rex_bits);
156 | }
157 | 
158 | 
159 | void Assembler::emit_optional_rex_32(Register rm_reg) {
160 |   if (rm_reg.high_bit()) emit(0x41);
161 | }
162 | 
163 | 
164 | void Assembler::emit_optional_rex_32(const Operand& op) {
165 |   if (op.rex_ != 0) emit(0x40 | op.rex_);
166 | }
167 | 
168 | 
169 | Address Assembler::target_address_at(Address pc) {
170 |   return Memory::int32_at(pc) + pc + 4;
171 | }
172 | 
173 | 
174 | void Assembler::set_target_address_at(Address pc, Address target) {
175 |   Memory::int32_at(pc) = static_cast<int32_t>(target - pc - 4);
176 |   CPU::FlushICache(pc, sizeof(int32_t));
177 | }
178 | 
179 | // -----------------------------------------------------------------------------
180 | // Implementation of Operand
181 | 
182 | void Operand::set_modrm(int mod, Register rm_reg) {
183 |   ASSERT(is_uint2(mod));
184 |   buf_[0] = mod << 6 | rm_reg.low_bits();
185 |   // Set REX.B to the high bit of rm.code().
186 |   rex_ |= rm_reg.high_bit();
187 | }
188 | 
189 | 
190 | void Operand::set_sib(ScaleFactor scale, Register index, Register base) {
191 |   ASSERT(len_ == 1);
192 |   ASSERT(is_uint2(scale));
193 |   ASSERT(!uses_reloc_data());   // Note supported yet.
194 |   // Use SIB with no index register only for base rsp or r12. Otherwise we
195 |   // would skip the SIB byte entirely.
196 |   ASSERT(!index.is(rsp) || base.is(rsp) || base.is(r12));
197 |   buf_[1] = (scale << 6) | (index.low_bits() << 3) | base.low_bits();
198 |   rex_ |= index.high_bit() << 1 | base.high_bit();
199 |   len_ = 2;
200 | }
201 | 
202 | void Operand::set_disp8(int disp) {
203 |   ASSERT(is_int8(disp));
204 |   ASSERT(len_ == 1 || len_ == 2);
205 |   int8_t* p = reinterpret_cast<int8_t*>(&buf_[len_]);
206 |   *p = disp;
207 |   len_ += sizeof(int8_t);
208 | }
209 | 
210 | void Operand::set_disp32(int disp) {
211 |   ASSERT(len_ == 1 || len_ == 2);
212 |   int32_t* p = reinterpret_cast<int32_t*>(&buf_[len_]);
213 |   *p = disp;
214 |   len_ += sizeof(int32_t);
215 | }
216 | 
217 | 
218 | } }  // namespace rejit::internal
219 | 
220 | #endif  // REJIT_X64_ASSEMBLER_X64_INL_H_
221 | 


--------------------------------------------------------------------------------
/src/x64/macro-assembler-x64-inl.h:
--------------------------------------------------------------------------------
  1 | // Copyright (C) 2013 Alexandre Rames <alexandre@coreperf.com>
  2 | 
  3 | // rejit is free software: you can redistribute it and/or modify
  4 | // it under the terms of the GNU General Public License as published by
  5 | // the Free Software Foundation, either version 3 of the License, or
  6 | // (at your option) any later version.
  7 | //
  8 | // This program is distributed in the hope that it will be useful,
  9 | // but WITHOUT ANY WARRANTY; without even the implied warranty of
 10 | // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 11 | // GNU General Public License for more details.
 12 | //
 13 | // You should have received a copy of the GNU General Public License
 14 | // along with this program.  If not, see <http://www.gnu.org/licenses/>.
 15 | 
 16 | #ifndef REJIT_X64_MACRO_ASSEMBLER_X64_INL_H_
 17 | #define REJIT_X64_MACRO_ASSEMBLER_X64_INL_H_
 18 | 
 19 | namespace rejit {
 20 | namespace internal {
 21 | 
 22 | 
 23 | void MacroAssembler::PushRegisters(RegList regs) {
 24 |   int i;
 25 |   for (i = Register::kNumRegisters - 1; i >= 0; i--) {
 26 |     if (regs & (1 << i)) {
 27 |       push(Register::from_code(i));
 28 |     }
 29 |   }
 30 | }
 31 | 
 32 | 
 33 | void MacroAssembler::PopRegisters(RegList regs) {
 34 |   int i;
 35 |   for (i = 0; i < Register::kNumRegisters; i++) {
 36 |     if (regs & (1 << i)) {
 37 |       pop(Register::from_code(i));
 38 |     }
 39 |   }
 40 | }
 41 | 
 42 | 
 43 | // TODO: We generally don't care about all those registers and don't need to
 44 | // preserve them all.
 45 | void MacroAssembler::PushCallerSavedRegisters() {
 46 |   PushRegisters(kCallerSavedRegList);
 47 | }
 48 | 
 49 | 
 50 | void MacroAssembler::PopCallerSavedRegisters() {
 51 |   PopRegisters(kCallerSavedRegList);
 52 | }
 53 | 
 54 | 
 55 | void MacroAssembler::PushAllRegisters() {
 56 |   // All registers except rsp.
 57 |   PushRegisters(0xffff & ~rsp.bit());
 58 | }
 59 | 
 60 | 
 61 | void MacroAssembler::PopAllRegisters() {
 62 |   // All registers except rsp.
 63 |   PopRegisters(0xffff & ~rsp.bit());
 64 | }
 65 | 
 66 | 
 67 | // TODO(rames): Push and pop ah instread of rax.
 68 | void MacroAssembler::PushAllRegistersAndFlags() {
 69 |   PushAllRegisters();
 70 |   lahf();
 71 |   push(rax);
 72 | }
 73 | 
 74 | 
 75 | void MacroAssembler::PopAllRegistersAndFlags() {
 76 |   pop(rax);
 77 |   sahf();
 78 |   PopAllRegisters();
 79 | }
 80 | 
 81 | 
 82 | void MacroAssembler::debug_msg(const char *message) {
 83 |   if (FLAG_emit_debug_code) {
 84 |     msg(message);
 85 |   }
 86 | }
 87 | 
 88 | 
 89 | void MacroAssembler::Move(Register dst, Register src) {
 90 |   if (!dst.is(src)) {
 91 |     movq(dst, src);
 92 |   }
 93 | }
 94 | 
 95 | 
 96 | void MacroAssembler::MoveCharsFrom(Register dst, unsigned n, const char *location) {
 97 |   if (n == 0) return;
 98 |   int64_t chars = *reinterpret_cast<const int64_t*>(location);
 99 |   Move(dst, chars & FirstCharsMask(n));
100 | }
101 | 
102 | 
103 | void MacroAssembler::LoadCharsFrom(Register dst, unsigned n, const Operand& src) {
104 |   ASSERT(0 < n && n <= (unsigned)kPointerSize);
105 |   movq(dst, src);
106 |   Move(scratch, FirstCharsMask(n));
107 |   and_(dst, scratch);
108 | }
109 | 
110 | 
111 | void MacroAssembler::LoadCurrentChar(Register r) {
112 |   // TODO(rames): Issue when accessing at a memory limit, eg. end of page?
113 |   movq(r, Operand(string_pointer, 0));
114 | }
115 | 
116 | 
117 | void MacroAssembler::MemZero(const Operand& op, size_t size, Register zero,
118 |                              MemZeroOutputStatus out_status,
119 |                              MemZeroSizeHint size_hint) {
120 |   Register start = zero.is(scratch1) ? scratch2 : scratch1;
121 |   lea(start, op);
122 |   MemZero(start, size, zero, out_status, size_hint);
123 | }
124 | 
125 | 
126 | void MacroAssembler::inc_c(Register dst) {
127 |   if (char_size() == 1) {
128 |     incq(dst);
129 |   } else {
130 |     addq(dst, Immediate(char_size()));
131 |   }
132 | }
133 | void MacroAssembler::dec_c(Register dst) {
134 |   if (char_size() == 1) {
135 |     decq(dst);
136 |   } else {
137 |     subq(dst, Immediate(char_size()));
138 |   }
139 | }
140 | 
141 | 
142 | void MacroAssembler::Advance(unsigned n_chars,
143 |                              Direction direction,
144 |                              Register reg) {
145 |   if (direction == kForward) {
146 |     if (kCharSize == 1) {
147 |       incq(string_pointer);
148 |     } else {
149 |       addq(string_pointer, Immediate(kCharSize));
150 |     }
151 |   } else {
152 |     if (kCharSize == 1) {
153 |       decq(string_pointer);
154 |     } else {
155 |       subq(string_pointer, Immediate(kCharSize));
156 |     }
157 |   }
158 | }
159 | 
160 | 
161 | void MacroAssembler::AdvanceToEOS() {
162 |   movq(string_pointer, string_end);
163 | }
164 | 
165 | 
166 | } }  // namespace rejit::internal
167 | 
168 | #endif  // REJIT_X64_MACRO_ASSEMBLER_X64_INL_H_
169 | 


--------------------------------------------------------------------------------
/src/x64/macro-assembler-x64.h:
--------------------------------------------------------------------------------
  1 | // Copyright (C) 2013 Alexandre Rames <alexandre@coreperf.com>
  2 | // rejit is free software: you can redistribute it and/or modify
  3 | // it under the terms of the GNU General Public License as published by
  4 | // the Free Software Foundation, either version 3 of the License, or
  5 | // (at your option) any later version.
  6 | //
  7 | // This program is distributed in the hope that it will be useful,
  8 | // but WITHOUT ANY WARRANTY; without even the implied warranty of
  9 | // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 10 | // GNU General Public License for more details.
 11 | //
 12 | // You should have received a copy of the GNU General Public License
 13 | // along with this program.  If not, see <http://www.gnu.org/licenses/>.
 14 | 
 15 | #ifndef REJIT_X64_MACRO_ASSEMBLER_X64_H_
 16 | #define REJIT_X64_MACRO_ASSEMBLER_X64_H_
 17 | 
 18 | #include "assembler-x64.h"
 19 | #include "assembler-x64-inl.h"
 20 | #include "macro-assembler-base.h"
 21 | 
 22 | namespace rejit {
 23 | namespace internal {
 24 | 
 25 | 
 26 | const Register result_matches = rbx;
 27 | const Register ring_index = r12;
 28 | 
 29 | const Register string_base = r13;
 30 | const Register string_pointer = r14;
 31 | // Points past the last character of the string not including '\0'. Ie. points
 32 | // to '\0' for '\0' terminated strings or just after the last character.
 33 | // Memory at and after string_end should not be accessed.
 34 | const Register string_end = r15;
 35 | const Operand current_char = Operand(string_pointer, 0);
 36 | const Operand next_char = Operand(string_pointer, kCharSize);
 37 | const Operand previous_char = Operand(string_pointer, -kCharSize);
 38 | const Operand current_chars = Operand(string_pointer, 0);
 39 | 
 40 | static inline int CalleeSavedRegsSize() {
 41 |   if (FLAG_emit_debug_code) {
 42 |     return kCalleeSavedRegsSize;
 43 |   } else {
 44 |     return kUsedCalleeSavedRegsSize;
 45 |   }
 46 | }
 47 | 
 48 | const int kStateInfoSize = 5 * kPointerSize;
 49 | // Next starting position for fast forwarding.
 50 | const Operand ff_position    (rbp, -CalleeSavedRegsSize() - 1 * kPointerSize);
 51 | // State from which FF thinks there may be a potential match.
 52 | const Operand ff_found_state (rbp, -CalleeSavedRegsSize() - 2 * kPointerSize);
 53 | // Position of the end of the match when matching the regexp backward from the
 54 | // ff_element.
 55 | const Operand backward_match (rbp, -CalleeSavedRegsSize() - 3 * kPointerSize);
 56 | // Position of the end of the match when matching the regexp forward from the
 57 | // ff_element.
 58 | const Operand forward_match  (rbp, -CalleeSavedRegsSize() - 4 * kPointerSize);
 59 | // Used when looking for multiple matches (kMatchAll) to indicate the end of the
 60 | // previous match.
 61 | const Operand last_match_end (rbp, -CalleeSavedRegsSize() - 5 * kPointerSize);
 62 | 
 63 | const Register mscratch = r8;
 64 | const Register scratch = r9;
 65 | const Register scratch1 = r9;
 66 | const Register scratch2 = r10;
 67 | const Register scratch3 = r11;
 68 | 
 69 | 
 70 | class MacroAssembler : public MacroAssemblerBase {
 71 |  public:
 72 |   MacroAssembler();
 73 |   // TODO(ajr): Check destructor need.
 74 | 
 75 |   inline void PushRegisters(RegList regs);
 76 |   inline void PopRegisters(RegList regs);
 77 | 
 78 |   // Currently following the System V ABI only.
 79 |   inline void PushCallerSavedRegisters();
 80 |   inline void PopCallerSavedRegisters();
 81 |   void PushCalleeSavedRegisters();
 82 |   void PopCalleeSavedRegisters();
 83 | 
 84 |   inline void PushAllRegisters();
 85 |   inline void PopAllRegisters();
 86 |   inline void PushAllRegistersAndFlags();
 87 |   inline void PopAllRegistersAndFlags();
 88 | 
 89 |   void CallCppPrepareStack();
 90 |   void CallCpp(Address address);
 91 | 
 92 |   void Move(Register dst, uint64_t value);
 93 |   inline void Move(Register dst, Register src);
 94 | 
 95 |   inline void MoveCharsFrom(Register dst, unsigned n, const char* chars);
 96 |   inline void LoadCharsFrom(Register dst, unsigned n, const Operand& src);
 97 | 
 98 |   inline void LoadCurrentChar(Register r);
 99 | 
100 |   void MaskFirstChars(unsigned n_chars, Register dst);
101 | 
102 |   // Variable width helpers.
103 |   void mov(unsigned width, Register dst, const Operand&);
104 |   void mov_truncated(unsigned width, Register dst, const Operand&);
105 | 
106 |   // Variable width compare.
107 |   // Accesses 2^log2(width) bytes at dst.
108 |   template <class Type>
109 |     void cmp_truncated_(unsigned width, Register dst, Type src) {
110 |       if (width == 0) return;
111 |       if (width == 1) {
112 |         cmpb(dst, src);
113 |       } else if (width < 4) {
114 |         cmpw(dst, src);
115 |       } else if (width < 8) {
116 |         cmpl(dst, src);
117 |       } else {
118 |         cmpq(dst, src);
119 |       }
120 |     }
121 |   inline void cmp_truncated(unsigned width, Register dst, Register src) {
122 |     return cmp_truncated_<Register>(width, dst, src);
123 |   }
124 |   inline void cmp_truncated(unsigned width, Register dst, const Operand& src) {
125 |     return cmp_truncated_<const Operand&>(width, dst, src);
126 |   }
127 |   void cmp_truncated(unsigned width, const Operand& dst, int64_t src);
128 |   // Variable width compare.
129 |   // Guarantees that no memory accesses from [dst + width] will occur.
130 |   void cmp_safe(unsigned width, Condition cond, Operand dst, int64_t src,
131 |                 Label* on_no_match = NULL);
132 |   void cmp_safe(unsigned width, Condition cond, Operand dst, Register src,
133 |                 Label* on_no_match = NULL);
134 |   // Variable width compare.
135 |   // Accesses 2^ceiling(log2(width)) bytes at dst.
136 |   void cmp(unsigned width, const Operand& dst, int64_t src);
137 |   void cmp(unsigned width, const Operand& dst, Register src);
138 | 
139 |   void movdq(XMMRegister dst, uint64_t high, uint64_t low);
140 |   void movdqp(XMMRegister dst, const char* chars, unsigned n_chars);
141 | 
142 |   // Clear a memory region.
143 |   // The start and end (and size) must be 8-bytes aligned.
144 |   // start must point at a strictly lower address than the end register.
145 |   enum MemZeroOutputStatus {
146 |     // On output, the start and end register point to the higher address.
147 |     AtLowAddress,
148 |     // On output, the start and end register point to the lower address.
149 |     AtHighAddress,
150 |   };
151 |   // Hint about the size of the copy.
152 |   enum MemZeroSizeHint {
153 |     Small,
154 |     Big,
155 |   };
156 |   void MemZero(Register start, Register end, Register zero = no_reg,
157 |                MemZeroOutputStatus = AtHighAddress, MemZeroSizeHint = Small);
158 |   inline void MemZero(const Operand& start, size_t size, Register zero = no_reg,
159 |                MemZeroOutputStatus = AtHighAddress, MemZeroSizeHint = Small);
160 |   void MemZero(Register start, size_t size, Register zero = no_reg,
161 |                MemZeroOutputStatus = AtHighAddress, MemZeroSizeHint = Small);
162 | 
163 |   // Increment or decrement by the size of a character.
164 |   inline void inc_c(Register dst);
165 |   inline void dec_c(Register dst);
166 | 
167 |   // Advance the register by a number of characters in the given direction.
168 |   inline void Advance(unsigned, Direction, Register);
169 |   inline void AdvanceToEOS();
170 | 
171 |   // Debug helpers ---------------------------------------------------
172 | 
173 |   // All debug helpers must preserve registers and flags.
174 | 
175 |   // Print a message.
176 |   void msg(const char* message);
177 |   inline void debug_msg(const char* message);
178 |   void debug_msg(Condition cond, const char* message);
179 | 
180 |   // Assert from assembly.
181 |   // Define a macro for convenience.
182 |   void asm_assert_(Condition cond, const char* file, int line, const char* description);
183 | 
184 |   // Print a message and stop.
185 |   void stop(const char* messgae);
186 |   void stop(Condition cond, const char* messgae);
187 | #define asm_assert(cond, description) asm_assert_(cond, __FILE__, __LINE__, description)
188 | };
189 | 
190 | 
191 | } }  // namespace rejit::internal
192 | 
193 | #endif  // REJIT_X64_MACRO_ASSEMBLER_X64_H_
194 | 
195 | 


--------------------------------------------------------------------------------
/tools/analysis/SConscript:
--------------------------------------------------------------------------------
 1 | import os
 2 | import sys
 3 | import subprocess
 4 | 
 5 | from os.path import join
 6 | from subprocess import PIPE
 7 | 
 8 | # Import rejit utils.
 9 | dir_root = os.path.realpath('.')
10 | while 'SConstruct' not in os.listdir(dir_root):
11 |   dir_root = os.path.realpath(join(dir_root, '..'))
12 | sys.path.insert(0, join(dir_root, 'tools'))
13 | import utils
14 | 
15 | # These are exported from SConstruct and further passed down to SConscript
16 | # files.
17 | Import('env librejit')
18 | 
19 | libs = [librejit]
20 | if 'os' in env and env['os'] == 'macos':
21 |   conf = Configure(env)
22 |   if not conf.CheckLib('argp'):
23 |     print utils.help_messages['argp']
24 |     Exit(1)
25 |   env = conf.Finish()
26 |   libs += ['argp']
27 | 
28 | env['CCFLAGS'] += ['-DMOD_FLAGS']
29 | analyze_CCFLAGS = env['CCFLAGS']
30 | analyze_CXXFLAGS = env['CXXFLAGS']
31 | analyze_CPPPATH = env['CPPPATH']
32 | 
33 | compinfo = env.Program('compinfo', 'compinfo.cc',
34 |                        LIBS=libs,
35 |                        CCFLAGS=analyze_CCFLAGS,
36 |                        CXXFLAGS=analyze_CXXFLAGS,
37 |                        CPPPATH=analyze_CPPPATH)
38 | env.Alias('compinfo', compinfo)
39 | 


--------------------------------------------------------------------------------
/tools/analysis/compinfo.cc:
--------------------------------------------------------------------------------
  1 | // Copyright (C) 2014 Alexandre Rames <alexandre@coreperf.com>
  2 | // rejit is free software: you can redistribute it and/or modify
  3 | // it under the terms of the GNU General Public License as published by
  4 | // the Free Software Foundation, either version 3 of the License, or
  5 | // (at your option) any later version.
  6 | //
  7 | // This program is distributed in the hope that it will be useful,
  8 | // but WITHOUT ANY WARRANTY; without even the implied warranty of
  9 | // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 10 | // GNU General Public License for more details.
 11 | //
 12 | // You should have received a copy of the GNU General Public License
 13 | // along with this program.  If not, see <http://www.gnu.org/licenses/>.
 14 | 
 15 | #include <assert.h>
 16 | #include <argp.h>
 17 | #include <string.h>
 18 | 
 19 | #include "rejit.h"
 20 | #include "flags.h"
 21 | 
 22 | using namespace rejit;
 23 | 
 24 | 
 25 | // Start the enum from the latest argp key used.
 26 | enum rejit_flags_option_keys {
 27 |   // Hope it does not collide with other keys.
 28 |   base_rejit_flag_key = 0x7BAD,
 29 | #define ENUM_KEYS(flag_name, r, d) flag_name##_key,
 30 |   REJIT_FLAGS_LIST(ENUM_KEYS)
 31 | #undef ENUM_KEYS
 32 |   after_last_rejit_key,
 33 |   first_rejit_flag_key = base_rejit_flag_key + 1
 34 | };
 35 | #define REJIT_FLAG_OFFSET(flag_name) (flag_name##_key - first_rejit_flag_key)
 36 | 
 37 | struct arguments {
 38 |   char *regexp;
 39 |   MatchType match_type;
 40 |   int  rejit_flags;
 41 |   bool print_all;
 42 | };
 43 | 
 44 | struct argp_option options[] =
 45 | {
 46 |   {"match_type" , 'm' , "all"  , OPTION_ARG_OPTIONAL , "Matching type. One of 'full', 'anywhere', 'first', 'all'. Default: 'all'"},
 47 | #define FLAG_OPTION(flag_name, r, d) \
 48 |   {#flag_name , flag_name##_key , FLAG_##flag_name ? "1" : "0"   , OPTION_ARG_OPTIONAL , "0 to disable, 1 to enable."},
 49 |   REJIT_FLAGS_LIST(FLAG_OPTION)
 50 | #undef FLAG_OPTION
 51 |   {"print_all" , 'p' , ""  , OPTION_ARG_OPTIONAL , "Force all --print* options to be set."},
 52 |   {0}
 53 | };
 54 | 
 55 | char args_doc[] = "regexp";
 56 | char doc[] =
 57 | "Disassemble the code generated for the specified regexp and match type.\n" ;
 58 | const char *argp_program_bug_address = "<alexandre@coreperf.com>";
 59 | error_t parse_opt(int key, char *arg, struct argp_state *state);
 60 | struct argp argp = {options, parse_opt, args_doc, doc};
 61 | 
 62 | 
 63 | error_t parse_opt(int key, char *arg, struct argp_state *state) {
 64 |   struct arguments *arguments = reinterpret_cast<struct arguments*>(state->input);
 65 |   switch (key) {
 66 |     case 'm': {
 67 |       if (!arg)
 68 |         break;
 69 |       if (strcmp("full", arg) == 0) {
 70 |         arguments->match_type = kMatchFull;
 71 |       } else if (strcmp("anywhere", arg) == 0) {
 72 |         arguments->match_type = kMatchAnywhere;
 73 |       } else if (strcmp("first", arg) == 0) {
 74 |         arguments->match_type = kMatchFirst;
 75 |       } else if (strcmp("all", arg) == 0) {
 76 |         arguments->match_type = kMatchAll;
 77 |       } else {
 78 |         printf("ERROR: Invalid match type\n.");
 79 |         argp_usage(state);
 80 |       }
 81 |       break;
 82 |     }
 83 | 
 84 |     case 'p': {
 85 |       arguments->print_all = true;
 86 |       break;
 87 |     }
 88 | 
 89 | #ifdef BENCH_ENGINE_REJIT
 90 | #define FLAG_CASE(flag_name, r, d)                                             \
 91 |     case flag_name##_key: {                                                    \
 92 |       if (arg) {                                                               \
 93 |         unsigned v;                                                            \
 94 |         v = stol(arg);                                                         \
 95 |         assert(v == 0 || v == 1);                                              \
 96 |         arguments->rejit_flags |= v << REJIT_FLAG_OFFSET(flag_name);           \
 97 |       }                                                                        \
 98 |       break;                                                                   \
 99 |     }
100 |     REJIT_FLAGS_LIST(FLAG_CASE)
101 | #undef FLAG_CASE
102 | #endif
103 | 
104 |     case ARGP_KEY_ARG:
105 |       if (state->arg_num >= 1) {
106 |         argp_usage(state);
107 |       }
108 |       arguments->regexp = arg;
109 |       break;
110 |     case ARGP_KEY_END:
111 |       if (state->arg_num < 1) {
112 |         argp_usage(state);
113 |       }
114 |       break;
115 |     default:
116 |       return ARGP_ERR_UNKNOWN;
117 |     }
118 |   return 0;
119 | }
120 | 
121 | 
122 | void handle_arguments(struct arguments *arguments,
123 |                       struct argp *argp,
124 |                       int argc,
125 |                       char *argv[]) {
126 |   arguments->regexp = NULL;
127 |   arguments->match_type = kMatchAll;
128 |   arguments->rejit_flags = 0;
129 |   arguments->print_all = false;
130 | 
131 | #ifdef BENCH_ENGINE_REJIT
132 | #define SET_FLAG_DEFAULT(flag_name, r, d)                                      \
133 |   arguments->rejit_flags |= FLAG_##flag_name << REJIT_FLAG_OFFSET(flag_name);
134 |   arguments->rejit_flags = 0;
135 |   REJIT_FLAGS_LIST(SET_FLAG_DEFAULT)
136 | #undef SET_FLAG_DEFAULT
137 | #endif
138 | 
139 |   argp_parse(argp, argc, argv, 0, 0, arguments);
140 | 
141 |   if (arguments->regexp == NULL || arguments->regexp[0] == '\0') {
142 |     printf("ERROR: Cannot test an empty regular expression.\n");
143 |     argp_usage(NULL);
144 |   }
145 | 
146 |   // Set rejit flags according to the arguments.
147 | #define MAYBE_SET_REJIT_FLAG(flag_name, r, d)                                  \
148 |   SET_FLAG(flag_name,                                                          \
149 |            arguments->rejit_flags & (1 << REJIT_FLAG_OFFSET(flag_name)));
150 |   REJIT_FLAGS_LIST(MAYBE_SET_REJIT_FLAG)
151 | #undef MAYBE_SET_REJIT_FLAG
152 | 
153 |   if (arguments->print_all) {
154 | #define SET_REJIT_FLAG(flag_name, r, d)                                        \
155 |   SET_FLAG(flag_name, 1);
156 |     REJIT_PRINT_FLAGS_LIST(SET_REJIT_FLAG)
157 | #undef SET_REJIT_FLAG
158 |   }
159 | }
160 | 
161 | 
162 | 
163 | int main(int argc, char *argv[]) {
164 |   struct arguments arguments;
165 |   handle_arguments(&arguments, &argp, argc, argv);
166 | 
167 |   // Ensure that compilation will dump the generated code.
168 |   SET_FLAG(dump_code, 1);
169 |   rejit::Regej re(arguments.regexp);
170 |   bool success = re.Compile(arguments.match_type);
171 | 
172 |   if (!success)
173 |     printf("%s\n", rejit_status_string);
174 | 
175 |   return !success;
176 | }
177 | 


--------------------------------------------------------------------------------
/tools/benchmarks/SConscript:
--------------------------------------------------------------------------------
 1 | import os
 2 | import sys
 3 | import subprocess
 4 | 
 5 | from os.path import join
 6 | from subprocess import PIPE
 7 | 
 8 | # Import rejit utils.
 9 | dir_root = os.path.realpath('.')
10 | while 'SConstruct' not in os.listdir(dir_root):
11 |   dir_root = os.path.realpath(join(dir_root, '..'))
12 | sys.path.insert(0, join(dir_root, 'tools'))
13 | import utils
14 | 
15 | # These are exported from SConstruct and further passed down to SConscript
16 | # files.
17 | Import('env librejit')
18 | 
19 | flot_remote = 'git://github.com/flot/flot.git'
20 | flot_dir = join(utils.dir_benchmarks_resources_html, 'flot')
21 | def flot_js_checkout(target, source, env):
22 |   print "Checking for javascript flot... ",
23 |   if os.path.exists(flot_dir) and os.path.isdir(flot_dir):
24 |     print "ok."
25 |   else:
26 |     print "not found."
27 |     utils.command_assert(['git', 'clone', flot_remote, flot_dir], "Failed to clone flot. Aborting build.")
28 | flot_js = env.Command(flot_dir, '', flot_js_checkout)
29 | env.Alias('flot_js', flot_js)
30 | 
31 | 
32 | SConscript('engines/pcre/SConscript', exports='env')
33 | SConscript('engines/re2/SConscript', exports='env')
34 | SConscript('engines/rejit/SConscript', exports='env librejit')
35 | SConscript('engines/v8/SConscript', exports='env')
36 | 


--------------------------------------------------------------------------------
/tools/benchmarks/engines/bench_engine.cc:
--------------------------------------------------------------------------------
  1 | #include "bench_engine.h"
  2 | 
  3 | #include <algorithm>
  4 | #include <iomanip>
  5 | #include <iostream>
  6 | #include <string>
  7 | 
  8 | #include <string.h>
  9 | #include <fcntl.h>
 10 | #include <sys/mman.h>
 11 | #include <sys/stat.h>
 12 | 
 13 | 
 14 | struct argp_option options[] =
 15 | {
 16 |   {"file"       , 'f' , ""     , OPTION_ARG_OPTIONAL , "Source file. If none provided, use a randomly generated characters."},
 17 |   {"size"       , 's' , "65536", OPTION_ARG_OPTIONAL , "Comma-separated list of text sizes."},
 18 |   {"iterations" , 'i' , "1000" , OPTION_ARG_OPTIONAL , "Number of iterations to run."},
 19 |   {"low_char"   , 'l' , "0"    , OPTION_ARG_OPTIONAL , "When the match source is random text, the low character of the range of characters composing the matched text."},
 20 |   {"high_char"  , 'h' , "z"    , OPTION_ARG_OPTIONAL , "When the match source is random text, the high character of the range of characters composing the matched text."},
 21 | #ifdef BENCH_ENGINE_REJIT
 22 |   // Convenient access to rejit flags.
 23 | #define FLAG_OPTION(flag_name, r, d) \
 24 |   {#flag_name , flag_name##_key , FLAG_##flag_name ? "1" : "0"   , OPTION_ARG_OPTIONAL , "0 to disable, 1 to enable."},
 25 |   REJIT_FLAGS_LIST(FLAG_OPTION)
 26 | #undef FLAG_OPTION
 27 | #endif
 28 | #define RUN_WORST_CASE_KEY (after_last_rejit_key + 1)
 29 |   {"run_worst_case" , RUN_WORST_CASE_KEY , "1", OPTION_ARG_OPTIONAL , "Run the benchmarks for the worst case. 0 to disable, 1 to enable."},
 30 |   {0}
 31 | };
 32 | 
 33 | static int on_off_from_str(int *res, const char *str) {
 34 |   if (0 == strcmp(str, "on") ||
 35 |       0 == strcmp(str, "true") ||
 36 |       0 == strcmp(str, "1")) {
 37 |     *res = 1;
 38 |     return 0;
 39 |   }
 40 |   if (0 == strcmp(str, "off") ||
 41 |       0 == strcmp(str, "false") ||
 42 |       0 == strcmp(str, "0")) {
 43 |     *res = 0;
 44 |     return 0;
 45 |   }
 46 |   return 1;
 47 | }
 48 | 
 49 | 
 50 | char doc[] =
 51 | "\n"
 52 | "Benchmark regular expression engine.\n"
 53 | "\n"
 54 | "Output: processing speed in bytes/s (<size of text matched> / <time to match>)\n"
 55 | "\t<worst speed> (1 run for 1 compilation)\n"
 56 | "\t<amortised speed> (<--iterations=?> runs for 1 compilation)\n"
 57 | "\t<best speed> (without considering compilation time)\n";
 58 | 
 59 | char args_doc[] = "regexp";
 60 | 
 61 | const char *argp_program_bug_address = "<alexandre@coreperf.com>";
 62 | 
 63 | struct argp argp = {options, parse_opt, args_doc, doc};
 64 | 
 65 | 
 66 | error_t parse_opt(int key, char *arg, struct argp_state *state) {
 67 |   struct arguments *arguments = reinterpret_cast<struct arguments*>(state->input);
 68 |   switch (key) {
 69 |     case 'f':
 70 |       arguments->file = arg;
 71 |       break;
 72 |     case 's': {
 73 |       size_t pos;
 74 |       size_t val;
 75 |       if (!arg)
 76 |         break;
 77 |       while (arg[0] != '\0') {
 78 |         val = stoll(arg, &pos);
 79 |         if (pos == 0)
 80 |           break;
 81 |         arguments->sizes.push_back(val);
 82 |         arg += pos + (arg[pos] == ',');
 83 |       }
 84 | 
 85 |       if (arguments->sizes.size() == 0 || arg[0] != '\0')
 86 |         error("Invalid sizes arguments.");
 87 | 
 88 |       break;
 89 |     }
 90 |     case 'i':
 91 |       if (arg) {
 92 |         arguments->iterations = stol(arg);
 93 |         if (arguments->iterations == 0) {
 94 |           error("The number of iterations to run must be greater than 0.");
 95 |         }
 96 |       }
 97 |       break;
 98 |     case 'l':
 99 |       arguments->low_char = arg[0];
100 |       break;
101 |     case 'h':
102 |       arguments->high_char = arg[0];
103 |       break;
104 | 
105 |     case RUN_WORST_CASE_KEY:
106 |       if (on_off_from_str(&arguments->run_worst_case, arg)) {
107 |         error("Invalid value for option 'run_worst_case'. "
108 |               "Expected one of (on|true|1|off|false|0).\n");
109 |       }
110 |       break;
111 | 
112 | #ifdef BENCH_ENGINE_REJIT
113 | #define FLAG_CASE(flag_name, r, d)                                             \
114 |     case flag_name##_key: {                                                    \
115 |       if (arg) {                                                               \
116 |         unsigned v;                                                            \
117 |         v = stol(arg);                                                         \
118 |         assert(v == 0 || v == 1);                                              \
119 |         arguments->rejit_flags |= v << REJIT_FLAG_OFFSET(flag_name);           \
120 |       }                                                                        \
121 |       break;                                                                   \
122 |     }
123 |     REJIT_FLAGS_LIST(FLAG_CASE)
124 | #undef FLAG_CASE
125 | #endif
126 | 
127 |     case ARGP_KEY_ARG:
128 |       if (state->arg_num >= 1) {
129 |         argp_usage(state);
130 |       }
131 |       arguments->args[state->arg_num] = arg;
132 |       break;
133 |     case ARGP_KEY_END:
134 |       if (state->arg_num < 1) {
135 |         argp_usage(state);
136 |       }
137 |       break;
138 |     default:
139 |       return ARGP_ERR_UNKNOWN;
140 |     }
141 |   return 0;
142 | }
143 | 
144 | 
145 | void handle_arguments(struct arguments *arguments,
146 |                       struct argp *argp,
147 |                       int argc,
148 |                       char *argv[]) {
149 |   arguments->file       = NULL;
150 |   arguments->iterations = 1000;
151 |   arguments->low_char   = 'a';
152 |   arguments->high_char  = 'z';
153 |   arguments->run_worst_case  = 1;
154 | 
155 | #ifdef BENCH_ENGINE_REJIT
156 | #define SET_FLAG_DEFAULT(flag_name, r, d)                                      \
157 |   arguments->rejit_flags |= FLAG_##flag_name << REJIT_FLAG_OFFSET(flag_name);
158 |   arguments->rejit_flags = 0;
159 |   REJIT_FLAGS_LIST(SET_FLAG_DEFAULT)
160 | #undef SET_FLAG_DEFAULT
161 | #endif
162 | 
163 |   argp_parse(argp, argc, argv, 0, 0, arguments);
164 | 
165 |   if (arguments->args[0][0] == '\0') {
166 |     error("Cannot test an empty regular expression.");
167 |   }
168 | 
169 | 
170 |   if (arguments->sizes.size() == 0)
171 |     arguments->sizes.push_back(65536);
172 |   else
173 |     sort(arguments->sizes.begin(), arguments->sizes.end());
174 | }
175 | 
176 | 
177 | void prepare_text(struct arguments *arguments, string *text) {
178 |   size_t max_text_size = arguments->sizes.back();
179 |   if (arguments->file) {
180 |     // Use the content of the specified file to fill text. If the file is
181 |     // smaller than the requested test size, copy it multiple times.
182 |     struct stat file_stats;
183 |     int fd = open(arguments->file, O_RDONLY);
184 |     fstat(fd, &file_stats);
185 |     char* file_text = (char*)mmap(NULL, file_stats.st_size,
186 |                                   PROT_READ, MAP_PRIVATE, fd, 0);
187 |     // Avoid the terminating character.
188 |     size_t file_text_size = file_stats.st_size - 1;
189 |     // Fill the text to search.
190 |     // If the source file is not big enough, loop over it.
191 |     size_t offset = 0;
192 |     size_t copy_size;
193 |     while (offset < max_text_size) {
194 |       copy_size = min(max_text_size - offset, file_text_size);
195 |       text->append(file_text, copy_size);
196 |       offset += copy_size;
197 |     }
198 | 
199 |     munmap(file_text, file_stats.st_size);
200 | 
201 |   } else {
202 |     text->resize(max_text_size);
203 |     for (size_t i = 0; i < max_text_size - 1; i++) {
204 |       text->at(i) = arguments->low_char +
205 |         (rand() % (arguments->high_char - arguments->low_char));
206 |     }
207 |   }
208 | }
209 | 
210 | 
211 | void print_results(vector<bench_res> *results, int print_worst_case)
212 | {
213 |   vector<bench_res>::iterator it;
214 |   unsigned max_width = strlen("text_size");
215 | 
216 |   for (it = results->begin(); it < results->end(); it++) {
217 |     unsigned width = 0;
218 |     size_t n = (*it).text_size;
219 |     while (n) {
220 |       n /= 10;
221 |       ++width;
222 |     }
223 |     max_width = max(max_width, width);
224 |   }
225 | 
226 |   cout << setw(max_width) << "text_size";
227 |   if (print_worst_case) {
228 |     cout << setw(16) << "worse";
229 |   }
230 |   cout << setw(16) << "amortised" <<
231 |     setw(16) << "best" << endl;
232 |   for (it = results->begin(); it < results->end(); it++) {
233 |   cout <<
234 |     setw(max_width) << (*it).text_size;
235 |     if (print_worst_case) {
236 |       cout << setw(16) << (*it).worse;
237 |     }
238 |     cout << setw(16) << (*it).amortised <<
239 |       setw(16) << (*it).best << endl;
240 |   }
241 | }
242 | 
243 | 
244 | double speed(struct timeval *t0, struct timeval *t1, size_t text_size, unsigned times) {
245 |   int64_t sec = t1->tv_sec - t0->tv_sec;
246 |   int64_t usec = t1->tv_usec - t0->tv_usec;
247 |   double time_usec = (double)usec + (double)sec * 1000000.0;
248 |   return ((double)text_size / time_usec) * 1000000.0 * (double)times;
249 | }
250 | 
251 | 
252 | void error(const char* message, int rc) {
253 |   printf("ERROR: %s\nExiting.\n", message);
254 |   exit(rc);
255 | }
256 | 


--------------------------------------------------------------------------------
/tools/benchmarks/engines/bench_engine.h:
--------------------------------------------------------------------------------
 1 | // Copyright (C) 2013 Alexandre Rames <alexandre@coreperf.com>
 2 | 
 3 | // rejit is free software: you can redistribute it and/or modify
 4 | // it under the terms of the GNU General Public License as published by
 5 | // the Free Software Foundation, either version 3 of the License, or
 6 | // (at your option) any later version.
 7 | //
 8 | // This program is distributed in the hope that it will be useful,
 9 | // but WITHOUT ANY WARRANTY; without even the implied warranty of
10 | // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
11 | // GNU General Public License for more details.
12 | //
13 | // You should have received a copy of the GNU General Public License
14 | // along with this program.  If not, see <http://www.gnu.org/licenses/>.
15 | 
16 | #ifndef BENCH_ENGINE_H_
17 | #define BENCH_ENGINE_H_
18 | 
19 | #include <iostream>
20 | #include <vector>
21 | 
22 | #include <argp.h>
23 | 
24 | 
25 | #ifdef BENCH_ENGINE_REJIT
26 | #include "rejit.h"
27 | #include "checks.h"
28 | #include "flags.h"
29 | using namespace rejit;
30 | #endif
31 | 
32 | 
33 | using namespace std;
34 | 
35 | 
36 | // Start the enum from the latest argp key used.
37 | enum rejit_flags_option_keys {
38 |   // Hope it does not collide with other keys.
39 |   base_rejit_flag_key = 0x7BAD,
40 | #ifdef BENCH_ENGINE_REJIT
41 | #define ENUM_KEYS(flag_name, r, d) flag_name##_key,
42 |   REJIT_FLAGS_LIST(ENUM_KEYS)
43 | #undef ENUM_KEYS
44 | #endif
45 |   after_last_rejit_key,
46 |   first_rejit_flag_key = base_rejit_flag_key + 1
47 | };
48 | #define REJIT_FLAG_OFFSET(flag_name) (flag_name##_key - first_rejit_flag_key)
49 | 
50 | 
51 | struct arguments {
52 |   char      *args[1];  // The regexp.
53 |   char      *file;
54 |   vector<size_t> sizes;
55 |   unsigned  iterations;
56 |   char      low_char;
57 |   char      high_char;
58 |   int       run_worst_case;
59 | #ifdef BENCH_ENGINE_REJIT
60 |   int  rejit_flags;
61 | #endif
62 | };
63 | 
64 | extern struct argp_option options[];
65 | extern char args_doc[];
66 | extern char doc[];
67 | extern struct argp argp;
68 | extern const char *argp_program_bug_address;
69 | 
70 | error_t parse_opt(int key, char *arg, struct argp_state *state);
71 | 
72 | void handle_arguments(struct arguments *arguments,
73 |                       struct argp *argp,
74 |                       int argc,
75 |                       char *argv[]);
76 | 
77 | void prepare_text(struct arguments *arguments, string *text);
78 | 
79 | struct bench_res {
80 |   size_t text_size;
81 |   double worse;
82 |   double amortised;
83 |   double best;
84 | };
85 | 
86 | void print_results(vector<bench_res> *results, int print_worst_case);
87 | 
88 | double speed(struct timeval *t0, struct timeval *t1, size_t text_size, unsigned times);
89 | 
90 | void error(const char* message, int rc = 1);
91 | 
92 | 
93 | #endif
94 | 


--------------------------------------------------------------------------------
/tools/benchmarks/engines/pcre/SConscript:
--------------------------------------------------------------------------------
  1 | # Copyright (C) 2013 Alexandre Rames <alexandre@coreperf.com>
  2 | 
  3 | # rejit is free software: you can redistribute it and/or modify
  4 | # it under the terms of the GNU General Public License as published by
  5 | # the Free Software Foundation, either version 3 of the License, or
  6 | # (at your option) any later version.
  7 | #
  8 | # This program is distributed in the hope that it will be useful,
  9 | # but WITHOUT ANY WARRANTY; without even the implied warranty of
 10 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 11 | # GNU General Public License for more details.
 12 | #
 13 | # You should have received a copy of the GNU General Public License
 14 | # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 15 | 
 16 | import os
 17 | import sys
 18 | import subprocess
 19 | 
 20 | from os.path import join
 21 | 
 22 | dir_pcre = os.path.realpath('.')
 23 | # Import rejit utils.
 24 | dir_root = dir_pcre
 25 | while 'SConstruct' not in os.listdir(dir_root):
 26 |   dir_root = os.path.realpath(join(dir_root, '..'))
 27 | sys.path.insert(0, join(dir_root, 'tools'))
 28 | import utils
 29 | 
 30 | 
 31 | # Import the environment to check for the os.
 32 | Import('env')
 33 | 
 34 | # We don't reuse the rejit environment to build third party engines.
 35 | local_env = Environment()
 36 | 
 37 | pcre_dir = join(utils.dir_benchmarks_engines, 'pcre/svn.pcre')
 38 | pcre_dir_svn = join(pcre_dir, '.svn')
 39 | pcre_dir_build = join(pcre_dir, 'build')
 40 | pcre_dir_lib = join(pcre_dir_build, 'lib')
 41 | pcre_path_lib = join(pcre_dir_lib, 'libpcre.a')
 42 | pcre_path_libcpp = join(pcre_dir_lib, 'libpcrecpp.a')
 43 | 
 44 | def pcre_checkout(target, source, env):
 45 |   utils.assert_available('svn')
 46 |   pcre_remote = "svn://vcs.exim.org/pcre/code/trunk"
 47 |   print "Checking for pcre... ",
 48 |   if os.path.exists(pcre_dir_svn) and os.path.isdir(pcre_dir_svn):
 49 |     print "ok."
 50 |   else:
 51 |     print "not found."
 52 |     utils.command_assert(['rm', '-rf', pcre_dir])
 53 |     print "Cloning pcre."
 54 |     utils.command_assert(['svn', 'checkout', pcre_remote, pcre_dir],
 55 |                          "Failed to clone pcre. Aborting build.")
 56 | 
 57 | def pcre_build_configure_fn(target, source, env):
 58 |   utils.command_assert(cwd_=pcre_dir, command=['./autogen.sh'])
 59 | 
 60 | def pcre_build_makefile_fn(target, source, env):
 61 |   pcre_command_configure = ['./configure', '--enable-jit', '--disable-shared', '--prefix=' + pcre_dir_build]
 62 |   utils.command_assert(cwd_=pcre_dir, command=pcre_command_configure)
 63 | 
 64 | def pcre_build_lib_fn(target, source, env):
 65 |   print "Building pcre. This may take some time."
 66 |   utils.command_assert(cwd_=pcre_dir, command=['make', '-j4', 'install'])
 67 | 
 68 | 
 69 | def FilteredGlob(pattern, omit=[]):
 70 |   return filter(lambda f: os.path.basename(f.path) not in omit, Glob(pattern))
 71 | 
 72 | pcre_repo = local_env.Command(pcre_dir_svn, '', pcre_checkout)
 73 | pcre_build_autogen = local_env.Command(join(pcre_dir, 'configure'), pcre_repo, pcre_build_configure_fn)
 74 | pcre_build_configure = local_env.Command(join(pcre_dir, 'Makefile'), pcre_build_autogen, pcre_build_makefile_fn)
 75 | pcre_build_make = local_env.Command([pcre_path_lib, pcre_path_libcpp], pcre_build_configure, pcre_build_lib_fn)
 76 | pcre_sources = FilteredGlob(os.path.join(pcre_dir, '*'), omit=['build'])
 77 | Depends(pcre_build_make, pcre_sources)
 78 | 
 79 | 
 80 | 
 81 | # Grab compilation environment variables.
 82 | local_env['CC'] = os.getenv('CC') or local_env['CC']
 83 | local_env['CXX'] = os.getenv('CXX') or local_env['CXX']
 84 | local_env['CCFLAGS'] = ['-Wall', '-pedantic', '-DBENCHTEST', '-O3']
 85 | local_env['CXXFLAGS'] = ['-std=c++11']
 86 | 
 87 | local_env['LIBPATH'] = env['LIBPATH'] + ':' + pcre_dir + ':' + pcre_dir_lib + ':'
 88 | 
 89 | 
 90 | dir_sconscript = Dir('.').srcnode().path
 91 | path_engine = os.path.join(dir_sconscript, 'engine')
 92 | 
 93 | libs = ['pcre', 'pcrecpp', 'pthread']
 94 | if 'os' in env and env['os'] == 'macos':
 95 |   conf = Configure(local_env)
 96 |   if not conf.CheckLib('argp'):
 97 |     error(utils.help_messages['argp'])
 98 |   local_env = conf.Finish()
 99 |   libs += ['argp']
100 | 
101 | bench_engine_CCFLAGS = env['CCFLAGS']
102 | bench_engine_CXXFLAGS = env['CXXFLAGS']
103 | bench_engine_CPPPATH = env['CPPPATH']
104 | bench_engine_obj_pcre = Object(source='../bench_engine.cc',
105 |                                target='bench_engine_pcre.o',
106 |                                CCFLAGS=bench_engine_CCFLAGS,
107 |                                CXXFLAGS=bench_engine_CXXFLAGS,
108 |                                CPPPATH=bench_engine_CPPPATH)
109 | pcre_engine_obj = local_env.Object('engine.cc', CPPPATH='svn.pcre/')
110 | Depends(pcre_engine_obj, pcre_build_make)
111 | pcre_engine = local_env.Program('engine',
112 |                                [pcre_engine_obj, bench_engine_obj_pcre],
113 |                                LIBS=libs, CPPPATH='svn.pcre/')
114 | local_env.Alias('pcre_engine', pcre_engine)
115 | 


--------------------------------------------------------------------------------
/tools/benchmarks/engines/pcre/engine.cc:
--------------------------------------------------------------------------------
  1 | // Copyright (C) 2013 Alexandre Rames <alexandre@coreperf.com>
  2 | 
  3 | // rejit is free software: you can redistribute it and/or modify
  4 | // it under the terms of the GNU General Public License as published by
  5 | // the Free Software Foundation, either version 3 of the License, or
  6 | // (at your option) any later version.
  7 | //
  8 | // This program is distributed in the hope that it will be useful,
  9 | // but WITHOUT ANY WARRANTY; without even the implied warranty of
 10 | // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 11 | // GNU General Public License for more details.
 12 | //
 13 | // You should have received a copy of the GNU General Public License
 14 | // along with this program.  If not, see <http://www.gnu.org/licenses/>.
 15 | 
 16 | #include <iostream>
 17 | #include <algorithm>
 18 | #include <stdio.h>
 19 | #include <stdlib.h>
 20 | #include <unistd.h>
 21 | #include <sys/time.h>
 22 | #include <string.h>
 23 | 
 24 | #include "pcrecpp.h"
 25 | #include "../bench_engine.h"
 26 | 
 27 | using namespace std;
 28 | 
 29 | // Argp configuration ----------------------------------------------------------
 30 | const char *argp_program_version = "pcre benchmark engine";
 31 | 
 32 | 
 33 | // Benchmark function ----------------------------------------------------------
 34 | 
 35 | void pcre_compilation_error() {
 36 |   printf("PCRE compilation error.");
 37 |   exit(1);
 38 | }
 39 | 
 40 | 
 41 | int pcre_match_all(pcrecpp::RE *re, string *text) {
 42 |   pcrecpp::StringPiece input(*text);
 43 |   string var;
 44 |   int value;
 45 |   while (re->FindAndConsume(&input, &var, &value)) {}
 46 |   return 0;
 47 | }
 48 | 
 49 | 
 50 | // We output 3 different speeds, in order:
 51 | //  - worst case speed (regexp compilation for each run)
 52 | //  - amortised speed (one compilation for 'times' run)
 53 | //  - best case speed (compilation time ignored)
 54 | int main(int argc, char *argv[]) {
 55 |   struct arguments arguments;
 56 |   handle_arguments(&arguments, &argp, argc, argv);
 57 |   char *regexp = arguments.args[0];
 58 | 
 59 | 
 60 |   string text;
 61 |   prepare_text(&arguments, &text);
 62 | 
 63 |   // Run ---------------------------------------------------
 64 | 
 65 |   vector<bench_res> results;
 66 | 
 67 |   sort(arguments.sizes.begin(), arguments.sizes.end());
 68 | 
 69 |   vector<size_t>::reverse_iterator rit;
 70 |   for (rit = arguments.sizes.rbegin(); rit < arguments.sizes.rend(); ++rit) {
 71 |     int rc = 0;
 72 |     bench_res res;
 73 |     struct timeval t0, t1, t2;
 74 |     size_t size = res.text_size = *rit;
 75 |     text.resize(size);
 76 | 
 77 |     if (arguments.run_worst_case) { // Measure worst case speed.
 78 |       gettimeofday(&t0, NULL);
 79 |       for (unsigned i = 0; i < arguments.iterations; i++) {
 80 |         pcrecpp::RE re(regexp);
 81 |         rc = pcre_match_all(&re, &text);
 82 |         if (rc)
 83 |           break;
 84 |       }
 85 |       gettimeofday(&t1, NULL);
 86 | 
 87 |       res.worse = rc != 0 ? 0 : speed(&t0, &t1, size, arguments.iterations);
 88 |     }
 89 | 
 90 |     { // Compute best and amortised speeds.
 91 |       pcrecpp::RE* re;
 92 | 
 93 |       gettimeofday(&t0, NULL);
 94 | 
 95 |       re = new pcrecpp::RE(regexp);
 96 |       if (!re)
 97 |         pcre_compilation_error();
 98 | 
 99 |       gettimeofday(&t1, NULL);
100 | 
101 |       for (unsigned i = 0; i < arguments.iterations; i++) {
102 |         rc = pcre_match_all(re, &text);
103 |         if (rc)
104 |           break;
105 |       }
106 | 
107 |       gettimeofday(&t2, NULL);
108 | 
109 |       delete re;
110 | 
111 |       if (rc != 0) {
112 |         res.amortised = 0;
113 |         res.best = 0;
114 |       } else {
115 |         res.amortised = speed(&t0, &t2, size, arguments.iterations);
116 |         res.best = speed(&t1, &t2, size, arguments.iterations);
117 |       }
118 | 
119 |       results.push_back(res);
120 |     }
121 |   }
122 | 
123 |   reverse(results.begin(), results.end());
124 |   print_results(&results, arguments.run_worst_case);
125 | 
126 |   return 0;
127 | }
128 | 
129 | 


--------------------------------------------------------------------------------
/tools/benchmarks/engines/re2/SConscript:
--------------------------------------------------------------------------------
  1 | # Copyright (C) 2013 Alexandre Rames <alexandre@coreperf.com>
  2 | 
  3 | # rejit is free software: you can redistribute it and/or modify
  4 | # it under the terms of the GNU General Public License as published by
  5 | # the Free Software Foundation, either version 3 of the License, or
  6 | # (at your option) any later version.
  7 | #
  8 | # This program is distributed in the hope that it will be useful,
  9 | # but WITHOUT ANY WARRANTY; without even the implied warranty of
 10 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 11 | # GNU General Public License for more details.
 12 | #
 13 | # You should have received a copy of the GNU General Public License
 14 | # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 15 | 
 16 | import os
 17 | import sys
 18 | import subprocess
 19 | 
 20 | from os.path import join
 21 | 
 22 | dir_re2 = os.path.realpath('.')
 23 | # Import rejit utils.
 24 | dir_root = dir_re2
 25 | while 'SConstruct' not in os.listdir(dir_root):
 26 |   dir_root = os.path.realpath(join(dir_root, '..'))
 27 | sys.path.insert(0, join(dir_root, 'tools'))
 28 | import utils
 29 | 
 30 | 
 31 | # Import the environment to check for the os.
 32 | Import('env')
 33 | 
 34 | # We don't reuse the rejit environment to build third party engines.
 35 | local_env = Environment()
 36 | 
 37 | re2_dir = join(utils.dir_benchmarks_engines, 're2/hg.re2')
 38 | re2_dir_hg = join(re2_dir, '.hg')
 39 | re2_path_lib = join(re2_dir, 'obj/libre2.a')
 40 | 
 41 | def re2_checkout(target, source, env):
 42 |   utils.assert_available('hg')
 43 |   re2_remote = "https://re2.googlecode.com/hg"
 44 |   print "Checking for re2... ",
 45 |   if os.path.exists(re2_dir_hg) and os.path.isdir(re2_dir_hg):
 46 |     print "ok."
 47 |   else:
 48 |     print "not found."
 49 |     utils.command_assert(['rm', '-rf', re2_dir])
 50 |     print "Cloning re2."
 51 |     utils.command_assert(['hg', 'clone', re2_remote, re2_dir],
 52 |                          "Failed to clone re2. Aborting build.")
 53 | 
 54 | def re2_build(target, source, env):
 55 |   print "Building re2. This may take some time."
 56 |   utils.command_assert(['make', '-C', re2_dir, '-j5'],
 57 |                        "Failed to build re2. Aborting build.")
 58 | 
 59 | re2_repo = local_env.Command(re2_dir_hg, '', re2_checkout)
 60 | re2_lib = local_env.Command(re2_path_lib, re2_repo, re2_build)
 61 | re2_sources = Glob(os.path.join(re2_dir, 're2', '*'))
 62 | Depends(re2_lib, re2_sources)
 63 | 
 64 | 
 65 | 
 66 | # Grab compilation environment variables.
 67 | local_env['CC'] = os.getenv('CC') or local_env['CC']
 68 | local_env['CXX'] = os.getenv('CXX') or local_env['CXX']
 69 | local_env['CCFLAGS'] = ['-Wall', '-pedantic', '-DBENCHTEST', '-O3']
 70 | local_env['CXXFLAGS'] = ['-std=c++11']
 71 | 
 72 | local_env['LIBPATH'] = env['LIBPATH'] + ':' + os.path.join(re2_dir, 'obj') + ':'
 73 | 
 74 | 
 75 | dir_sconscript = Dir('.').srcnode().path
 76 | path_engine = os.path.join(dir_sconscript, 'engine')
 77 | 
 78 | libs = ['re2', 'pthread']
 79 | if 'os' in env and env['os'] == 'macos':
 80 |   conf = Configure(local_env)
 81 |   if not conf.CheckLib('argp'):
 82 |     error(utils.help_messages['argp'])
 83 |   local_env = conf.Finish()
 84 |   libs += ['argp']
 85 | 
 86 | bench_engine_CCFLAGS = env['CCFLAGS']
 87 | bench_engine_CXXFLAGS = env['CXXFLAGS']
 88 | bench_engine_CPPPATH = env['CPPPATH']
 89 | bench_engine_obj_re2 = Object(source='../bench_engine.cc',
 90 |                               target='bench_engine_re2.o',
 91 |                               CCFLAGS=bench_engine_CCFLAGS,
 92 |                               CXXFLAGS=bench_engine_CXXFLAGS,
 93 |                               CPPPATH=bench_engine_CPPPATH)
 94 | re2_engine_obj = local_env.Object('engine.cc', CPPPATH='hg.re2/')
 95 | Depends(re2_engine_obj, re2_lib)
 96 | re2_engine = local_env.Program('engine',
 97 |                                [re2_engine_obj, bench_engine_obj_re2],
 98 |                                LIBS=libs, CPPPATH='hg.re2/')
 99 | local_env.Alias('re2_engine', re2_engine)
100 | 


--------------------------------------------------------------------------------
/tools/benchmarks/engines/re2/engine.cc:
--------------------------------------------------------------------------------
 1 | // Copyright (C) 2013 Alexandre Rames <alexandre@coreperf.com>
 2 | 
 3 | // rejit is free software: you can redistribute it and/or modify
 4 | // it under the terms of the GNU General Public License as published by
 5 | // the Free Software Foundation, either version 3 of the License, or
 6 | // (at your option) any later version.
 7 | //
 8 | // This program is distributed in the hope that it will be useful,
 9 | // but WITHOUT ANY WARRANTY; without even the implied warranty of
10 | // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
11 | // GNU General Public License for more details.
12 | //
13 | // You should have received a copy of the GNU General Public License
14 | // along with this program.  If not, see <http://www.gnu.org/licenses/>.
15 | 
16 | #include <iostream>
17 | #include <algorithm>
18 | #include <stdio.h>
19 | #include <stdlib.h>
20 | #include <unistd.h>
21 | #include <sys/time.h>
22 | #include <string.h>
23 | 
24 | #include "hg.re2/re2/re2.h"
25 | #include "../bench_engine.h"
26 | 
27 | using namespace std;
28 | 
29 | // Argp configuration ----------------------------------------------------------
30 | const char *argp_program_version = "re2 benchmark engine";
31 | 
32 | 
33 | // Benchmark function ----------------------------------------------------------
34 | // We output 3 different speeds, in order:
35 | //  - worst case speed (regexp compilation for each run)
36 | //  - amortised speed (one compilation for 'times' run)
37 | //  - best case speed (compilation time ignored)
38 | int main(int argc, char *argv[]) {
39 |   struct arguments arguments;
40 |   handle_arguments(&arguments, &argp, argc, argv);
41 |   char *regexp = arguments.args[0];
42 | 
43 | 
44 |   string text;
45 |   prepare_text(&arguments, &text);
46 | 
47 |   // Run ---------------------------------------------------
48 | 
49 |   vector<bench_res> results;
50 | 
51 |   sort(arguments.sizes.begin(), arguments.sizes.end());
52 | 
53 |   vector<size_t>::reverse_iterator rit;
54 |   for (rit = arguments.sizes.rbegin(); rit < arguments.sizes.rend(); ++rit) {
55 |     bench_res res;
56 |     struct timeval t0, t1, t2;
57 |     size_t size = res.text_size = *rit;
58 |     text.resize(size);
59 | 
60 |     if (arguments.run_worst_case) { // Measure worst case speed.
61 |       gettimeofday(&t0, NULL);
62 |       for (unsigned i = 0; i < arguments.iterations; i++) {
63 |         re2::StringPiece piece = text;
64 |         RE2 pattern(regexp);
65 |         while (RE2::FindAndConsume(&piece, pattern)) {}
66 |       }
67 |       gettimeofday(&t1, NULL);
68 | 
69 |       res.worse = speed(&t0, &t1, size, arguments.iterations);
70 |     }
71 | 
72 |     { // Compute best and amortised speeds.
73 |       gettimeofday(&t0, NULL);
74 |       RE2 pattern(regexp);
75 | 
76 |       gettimeofday(&t1, NULL);
77 | 
78 |       for (unsigned i = 0; i < arguments.iterations; i++) {
79 |         re2::StringPiece piece = text;
80 |         while (RE2::FindAndConsume(&piece, pattern)) {}
81 |       }
82 | 
83 |       gettimeofday(&t2, NULL);
84 | 
85 |       res.amortised = speed(&t0, &t2, size, arguments.iterations);
86 |       res.best = speed(&t1, &t2, size, arguments.iterations);
87 | 
88 |       results.push_back(res);
89 |     }
90 |   }
91 | 
92 |   reverse(results.begin(), results.end());
93 |   print_results(&results, arguments.run_worst_case);
94 | 
95 |   return 0;
96 | }
97 | 
98 | 


--------------------------------------------------------------------------------
/tools/benchmarks/engines/rejit/SConscript:
--------------------------------------------------------------------------------
 1 | # Copyright (C) 2013 Alexandre Rames <alexandre@coreperf.com>
 2 | 
 3 | # rejit is free software: you can redistribute it and/or modify
 4 | # it under the terms of the GNU General Public License as published by
 5 | # the Free Software Foundation, either version 3 of the License, or
 6 | # (at your option) any later version.
 7 | #
 8 | # This program is distributed in the hope that it will be useful,
 9 | # but WITHOUT ANY WARRANTY; without even the implied warranty of
10 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
11 | # GNU General Public License for more details.
12 | #
13 | # You should have received a copy of the GNU General Public License
14 | # along with this program.  If not, see <http://www.gnu.org/licenses/>.
15 | 
16 | import os
17 | import sys
18 | 
19 | from os.path import join
20 | 
21 | dir_rejit = os.path.realpath('.')
22 | # Import rejit utils.
23 | dir_root = dir_rejit
24 | while 'SConstruct' not in os.listdir(dir_root):
25 |   dir_root = os.path.realpath(join(dir_root, '..'))
26 | sys.path.insert(0, join(dir_root, 'tools'))
27 | import utils
28 | 
29 | 
30 | Import('env librejit')
31 | 
32 | dir_sconscript = Dir('.').srcnode().path 
33 | path_engine = os.path.join(dir_sconscript, 'engine')
34 | 
35 | libs = [librejit]
36 | if 'os' in env and env['os'] == 'macos':
37 |   conf = Configure(env)
38 |   if not conf.CheckLib('argp'):
39 |     print utils.help_messages['argp']
40 |     Exit(1)
41 |   env = conf.Finish()
42 |   libs += ['argp']
43 | 
44 | env['CCFLAGS'] += ['-DBENCH_ENGINE_REJIT', '-DMOD_FLAGS']
45 | bench_engine_CCFLAGS = env['CCFLAGS']
46 | bench_engine_CXXFLAGS = env['CXXFLAGS']
47 | bench_engine_CPPPATH = env['CPPPATH']
48 | bench_engine_obj_rejit = Object(source='../bench_engine.cc',
49 |                                 target='bench_engine_rejit.o',
50 |                                 CCFLAGS=bench_engine_CCFLAGS,
51 |                                 CXXFLAGS=bench_engine_CXXFLAGS,
52 |                                 CPPPATH=bench_engine_CPPPATH)
53 | 
54 | rejit_engine = env.Program('engine', ['engine.cc', bench_engine_obj_rejit], LIBS=libs)
55 | env.Alias('rejit_engine', rejit_engine)
56 | 


--------------------------------------------------------------------------------
/tools/benchmarks/engines/rejit/engine.cc:
--------------------------------------------------------------------------------
  1 | // Copyright (C) 2013 Alexandre Rames <alexandre@coreperf.com>
  2 | 
  3 | // rejit is free software: you can redistribute it and/or modify
  4 | // it under the terms of the GNU General Public License as published by
  5 | // the Free Software Foundation, either version 3 of the License, or
  6 | // (at your option) any later version.
  7 | //
  8 | // This program is distributed in the hope that it will be useful,
  9 | // but WITHOUT ANY WARRANTY; without even the implied warranty of
 10 | // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 11 | // GNU General Public License for more details.
 12 | //
 13 | // You should have received a copy of the GNU General Public License
 14 | // along with this program.  If not, see <http://www.gnu.org/licenses/>.
 15 | 
 16 | #include <iostream>
 17 | #include <stdio.h>
 18 | #include <stdlib.h>
 19 | #include <unistd.h>
 20 | #include <fcntl.h>
 21 | #include <sys/time.h>
 22 | #include <sys/mman.h>
 23 | #include <string.h>
 24 | #include <string>
 25 | 
 26 | #include "../bench_engine.h"
 27 | 
 28 | #include "rejit.h"
 29 | using namespace rejit;
 30 | 
 31 | using namespace std;
 32 | 
 33 | // TODO(rames): Merge the code for rejit and re2 engines.
 34 | 
 35 | // Argp configuration ----------------------------------------------------------
 36 | const char *argp_program_version = "rejit benchmark engine";
 37 | 
 38 | 
 39 | // Benchmark function ----------------------------------------------------------
 40 | // We output 3 different speeds, in order:
 41 | //  - worst case speed (regexp compilation for each run)
 42 | //  - amortised speed (one compilation for 'times' run)
 43 | //  - best case speed (compilation time ignored)
 44 | int main(int argc, char *argv[]) {
 45 |   struct arguments arguments;
 46 |   handle_arguments(&arguments, &argp, argc, argv);
 47 |   char *regexp = arguments.args[0];
 48 | 
 49 |   // Set rejit flags according to the arguments.
 50 | #define SET_REJIT_FLAG(flag_name, r, d)                                        \
 51 |   SET_FLAG(flag_name,                                                          \
 52 |            arguments.rejit_flags & (1 << REJIT_FLAG_OFFSET(flag_name)));
 53 | REJIT_FLAGS_LIST(SET_REJIT_FLAG)
 54 | #undef SET_REJIT_FLAG
 55 | 
 56 |   Regej test_regexp(regexp);
 57 |   if (test_regexp.status()) {
 58 |     printf("%s\n", rejit_status_string);
 59 |     error("Invalid regular expression.\n");
 60 |   }
 61 | 
 62 |   string text;
 63 |   prepare_text(&arguments, &text);
 64 | 
 65 |   // Run ---------------------------------------------------
 66 | 
 67 |   vector<bench_res> results;
 68 | 
 69 |   for (unsigned i = 0; i < arguments.sizes.size(); i++) {
 70 |     struct timeval t0, t1, t2;
 71 |     bench_res res;
 72 |     size_t size = res.text_size = arguments.sizes.at(i);
 73 | 
 74 |     if (arguments.run_worst_case) { // Measure worst case speed.
 75 |       vector<Match> matches;
 76 |       gettimeofday(&t0, NULL);
 77 |       for (unsigned i = 0; i < arguments.iterations; i++) {
 78 |         Regej re(regexp);
 79 |         re.MatchAll(text.c_str(), size, &matches);
 80 |       }
 81 |       gettimeofday(&t1, NULL);
 82 | 
 83 |       res.worse = speed(&t0, &t1, size, arguments.iterations);
 84 |     }
 85 | 
 86 |     { // Compute best and amortised speeds.
 87 |       vector<Match> matches;
 88 |       gettimeofday(&t0, NULL);
 89 |       Regej re(regexp);
 90 | 
 91 |       re.Compile(kMatchAll);
 92 | 
 93 |       gettimeofday(&t1, NULL);
 94 | 
 95 |       for (unsigned i = 0; i < arguments.iterations; i++) {
 96 |         re.MatchAll(text.c_str(), size, &matches);
 97 |       }
 98 | 
 99 |       gettimeofday(&t2, NULL);
100 | 
101 |       res.amortised = speed(&t0, &t2, size, arguments.iterations);
102 |       res.best = speed(&t1, &t2, size, arguments.iterations);
103 | 
104 |       results.push_back(res);
105 |     }
106 |   }
107 | 
108 |   print_results(&results, arguments.run_worst_case);
109 | 
110 |   return 0;
111 | }
112 | 
113 | 


--------------------------------------------------------------------------------
/tools/benchmarks/engines/v8/SConscript:
--------------------------------------------------------------------------------
 1 | # Copyright (C) 2013 Alexandre Rames <alexandre@coreperf.com>
 2 | 
 3 | # rejit is free software: you can redistribute it and/or modify
 4 | # it under the terms of the GNU General Public License as published by
 5 | # the Free Software Foundation, either version 3 of the License, or
 6 | # (at your option) any later version.
 7 | #
 8 | # This program is distributed in the hope that it will be useful,
 9 | # but WITHOUT ANY WARRANTY; without even the implied warranty of
10 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
11 | # GNU General Public License for more details.
12 | #
13 | # You should have received a copy of the GNU General Public License
14 | # along with this program.  If not, see <http://www.gnu.org/licenses/>.
15 | 
16 | import os
17 | import sys
18 | import subprocess
19 | 
20 | from os.path import join
21 | 
22 | dir_v8 = os.path.realpath('.')
23 | # Import rejit utils.
24 | dir_root = dir_v8
25 | while 'SConstruct' not in os.listdir(dir_root):
26 |   dir_root = os.path.realpath(join(dir_root, '..'))
27 | sys.path.insert(0, join(dir_root, 'tools'))
28 | import utils
29 | 
30 | 
31 | # Import the environment to check for the os.
32 | Import('env')
33 | 
34 | # We don't reuse the rejit environment to build third party engines.
35 | local_env = Environment()
36 | 
37 | v8_dir = join(utils.dir_benchmarks_engines, 'v8/git.v8')
38 | v8_git_dir = join(v8_dir, '.git')
39 | v8_engine_path = join(v8_dir, 'out/native/d8')
40 | 
41 | def v8_clone(target, source, env):
42 |   utils.assert_available('git')
43 |   v8_remote = "git://github.com/v8/v8.git"
44 |   print "Checking for v8... ",
45 |   if os.path.exists(v8_git_dir) and os.path.isdir(v8_git_dir):
46 |     print "ok."
47 |   else:
48 |     print "not found."
49 |     utils.command_assert(['rm', '-rf', v8_dir])
50 |     print "Cloning v8."
51 |     utils.command_assert(['git', 'clone', v8_remote, v8_dir],
52 |                          "Failed to clone v8. Aborting build.")
53 | 
54 | def v8_build(target, source, env):
55 |   print "Building v8. This may take some time."
56 |   print ' '.join(['make', '-C', v8_dir, '-j3', 'dependencies'])
57 |   utils.command_assert(['make', '-C', v8_dir, '-j3', 'dependencies'],
58 |                        "Failed to build v8 dependencies. Aborting build.")
59 |   print ' '.join(['make', '-C', v8_dir, '-j3', 'native'])
60 |   utils.command_assert(['make', '-C', v8_dir, '-j3', 'native'],
61 |                        "Failed to build v8. Aborting build.")
62 | 
63 | v8_repo = local_env.Command(v8_git_dir, '', v8_clone)
64 | v8_engine = local_env.Command(v8_engine_path, v8_repo, v8_build)
65 | v8_sources = Glob(os.path.join(v8_dir, 'src', '*')) + Glob(os.path.join(v8_dir, 'src', 'x64', '*')) + Glob(os.path.join(v8_dir, 'src', 'ia32', '*'))
66 | Depends(v8_engine, v8_sources)
67 | 
68 | local_env.Alias('v8_engine', v8_engine)
69 | 


--------------------------------------------------------------------------------
/tools/benchmarks/engines/v8/engine:
--------------------------------------------------------------------------------
  1 | #!/usr/bin/python
  2 | 
  3 | import os
  4 | from os.path import join, dirname, realpath
  5 | import sys
  6 | import subprocess
  7 | 
  8 | # Used to generate random characters.
  9 | import random
 10 | # Used to compute the scores.
 11 | import math
 12 | 
 13 | scriptpath = dirname(realpath(__file__))
 14 | dir_tools = join(scriptpath, '../../../')
 15 | sys.path.insert(0, dir_tools)
 16 | import utils
 17 | from utils import *
 18 | import gc
 19 | 
 20 | import argparse
 21 | help_description='''
 22 | Benchmark v8 regular expression engine.
 23 | Outputs the processing speed in bytes/s (<size of text matched> / <time to match>)
 24 | '''
 25 | parser = argparse.ArgumentParser(description=help_description)
 26 | parser.add_argument("regexp" , help="The regular expression to benchmark.")
 27 | parser.add_argument("--file"       ,            default=""   , help="Source file. If none provided, use a randomly generated characters.")
 28 | parser.add_argument("--iterations" , type=int , default=1000 , help="Number of iterations to run.")
 29 | parser.add_argument("--low_char"   ,            default="0"  , help="When the match source is random text , the low character of the range of characters composing the matched text.")
 30 | parser.add_argument("--high_char"  ,            default="z"  , help="When the match source is random text , the high character of the range of characters composing the matched text.")
 31 | parser.add_argument("--size"       , type=int , nargs='+' , default=65536, help="List of text sizes.")
 32 | 
 33 | path_d8 = join(dir_benchmarks_engines, 'v8/git.v8/out/native/d8')
 34 | 
 35 | # TODO: faster implementation?
 36 | def random_text(size, low = '0', high = 'z'):
 37 |   chars = ''.join([chr(i) for i in range(ord(low), ord(high)) if i not in [ord(';'), ord('"'), ord('\\')]])
 38 |   return ''.join(random.choice(chars) for x in range(size))
 39 | 
 40 | def run():
 41 |   args = parser.parse_args()
 42 | 
 43 |   # Decide where the benchmark files will be created.
 44 |   dir_testfiles = join(dir_benchmarks_engines, 'v8', 'test_files')
 45 |   if not os.path.exists(dir_testfiles):
 46 |     os.makedirs(dir_testfiles)
 47 | 
 48 |   # The parameters should be embedded in the file names.
 49 |   # Writing those files in python is slow, and this allows us to reuse them.
 50 |   text_file_path = join(dir_testfiles, 'text.' + args.low_char + args.high_char + str(max(args.size)) + '.js')
 51 |   if not os.path.exists(text_file_path):
 52 |     text_file = open(text_file_path, 'w')
 53 |     text = random_text(max(args.size), args.low_char, args.high_char)
 54 |     text_file.write(' var text = "%s"; ' % text)
 55 |     text_file.close()
 56 | 
 57 |   dic = {
 58 |       'regexp': args.regexp,
 59 |       'sizes': ','.join(map(str, args.size)),
 60 |       'iterations': args.iterations,
 61 |       'modifier': 'g'}
 62 |   run_file_path = join(dir_testfiles, 'run.js')
 63 |   run_file = open(run_file_path, 'w')
 64 |   run_file.write('''
 65 | function main() {
 66 |   var regexp     = /%(regexp)s/%(modifier)s;
 67 |   var sizes      = [%(sizes)s];
 68 |   var iterations = %(iterations)s;
 69 |   var run_iterations;
 70 |   var s;
 71 |   var t0, tdiff;
 72 | 
 73 |   if (sizes.length > 0) {
 74 |     print("text_size	speed");
 75 |   }
 76 |   for (var index in sizes) {
 77 |     tdiff = 0;
 78 |     // Run each benchmark for at least 700ms while not more than 1024 the
 79 |     // specified number of iterations.
 80 |     for (run_iterations = iterations;
 81 |          tdiff < 25 && run_iterations <= 1024 * iterations;
 82 |          run_iterations = run_iterations * 2) {
 83 |       s = text.substring(0, sizes[index]);
 84 |       t0 = new Date();
 85 |       for (var i = 0; i < run_iterations; i++) {
 86 |         s.match(regexp);
 87 |       }
 88 |       tdiff = new Date() - t0;
 89 |     }
 90 |     if (tdiff != 0) {
 91 |       print(sizes[index] + "	" + sizes[index] / tdiff * 1000 * run_iterations);
 92 |     }
 93 |   }
 94 | }
 95 | main();
 96 | ''' % dic)
 97 |   run_file.close()
 98 | 
 99 |   command_args = [path_d8, text_file_path, run_file_path]
100 |   p = subprocess.Popen(command_args , stdout=subprocess.PIPE)
101 |   ret = p.wait()
102 |   out = p.communicate()
103 |   outs = out[0].rstrip(' \n\r\t').lstrip(' \n\r\t')
104 |   if ret != 0 or outs == 'Infinity':
105 |     print '0'
106 |   else:
107 |     print outs
108 | 
109 | if __name__ == "__main__":
110 |   run()
111 | 


--------------------------------------------------------------------------------
/tools/benchmarks/gjrep.py:
--------------------------------------------------------------------------------
  1 | #!/usr/bin/python
  2 | # -*- coding: utf-8 -*-
  3 | 
  4 | import os
  5 | import sys
  6 | import argparse
  7 | import subprocess
  8 | 
  9 | 
 10 | # Import rejit utils.
 11 | dir_benchmarks = os.path.dirname(os.path.realpath(__file__))
 12 | dir_rejit = dir_benchmarks
 13 | while 'SConstruct' not in os.listdir(dir_rejit):
 14 |   dir_rejit = os.path.realpath(os.path.join(dir_rejit, '..'))
 15 | sys.path.insert(0, os.path.join(dir_rejit, 'tools'))
 16 | import utils
 17 | 
 18 | 
 19 | 
 20 | description = '''
 21 | Run grepping benchmarks.
 22 | grep and jrep must be in your path.'''
 23 | 
 24 | def validate_directory(path):
 25 |   if not os.path.isdir(path):
 26 |     raise argparse.ArgumentError("{0} invalid")
 27 |   return path
 28 | 
 29 | parser = argparse.ArgumentParser(description=description,
 30 |                                  formatter_class=argparse.ArgumentDefaultsHelpFormatter)
 31 | 
 32 | engines_available = ['grep', 'jrep', 'git grep']
 33 | engines_default = ['grep', 'jrep']
 34 | parser.add_argument('--engines', action='store', nargs='+',
 35 |                     choices=engines_available, default=engines_default,
 36 |                     help='''
 37 | Space-separated list of grep utilities to benchmark. (To benchmark `git grep`,
 38 | the target directory must be within the current git repo, and use quotes to
 39 | escape the space.)
 40 | ''')
 41 | parser.add_argument('-i', '--iterations', type=int, default=5,
 42 |                     help="Number of iterations to run for each benchmarks")
 43 | parser.add_argument('directory',
 44 |                     type=validate_directory,
 45 |                     help="Directory to recursively grep through"),
 46 | 
 47 | args = parser.parse_args()
 48 | 
 49 | 
 50 | 
 51 | 
 52 | regexps_easy = [
 53 |   "foobar",
 54 |   "éphémère",
 55 |   "unsigned",
 56 |   "A_quite_long_string_of_36_characters"
 57 | ]
 58 | regexps_alt = [
 59 |   "alternation|words",
 60 |   "NOTfound|foobar|typedef|éphémère",
 61 |   "typedef|definition|definitively"
 62 | ]
 63 | regexps_hard = [
 64 |   "[a-z]{3,10}.*gnu.*[a-z]{3,10}",
 65 |   "^\s*(void|unsigned|int)\s+func\s+\(",
 66 | ]
 67 | # Use underscores '_' to limit the number of matches for small digits.
 68 | regexps_alt_linear = [
 69 |   "NOTfound|foobar|rare|typedef",
 70 |   "1__",
 71 |   "1__|22_",
 72 |   "1__|22_|333",
 73 |   "1__|22_|333|4444",
 74 |   "1__|22_|333|4444|55555",
 75 |   "1__|22_|333|4444|55555|666666",
 76 |   "1__|22_|333|4444|55555|666666|7777777",
 77 |   "1__|22_|333|4444|55555|666666|7777777|88888888",
 78 |   "1__|22_|333|4444|55555|666666|7777777|88888888|999999999",
 79 | ]
 80 | 
 81 | regexps = regexps_easy + regexps_alt + regexps_hard + regexps_alt_linear
 82 | 
 83 | 
 84 | 
 85 | engines = [e for e in engines_available if e in args.engines]
 86 | engines_options_base = {
 87 |   'grep': ['--recursive', '--with-filename', '--line-number', '--extended-regexp'],
 88 |   'git grep': ['--untracked', '--extended-regexp'],
 89 |   'jrep': ['--recursive', '--with-filename', '--line-number']}
 90 | engines_options_extra = {
 91 |   'jrep': ['-j']
 92 | }
 93 | 
 94 | results = {}
 95 | 
 96 | 
 97 | 
 98 | 
 99 | def quotify(x):
100 |   return '"' + x + '"'
101 | 
102 | 
103 | 
104 | class TimeResult:
105 |   def __init__(self, time_output):
106 |     self.sys = 0
107 |     self.user = 0
108 |     self.real = 0
109 | 
110 |     outs = time_output.split('\n')
111 |     for line in outs:
112 |       if len(line) == 0:
113 |         continue
114 |       (label, value) = line.split()
115 |       if label == 'sys':
116 |         self.sys = value
117 |       if label == 'user':
118 |         self.user = value
119 |       if label == 'real':
120 |         self.real = value
121 | 
122 |   def print_label(self):
123 |     print 'sys: %s\tuser: %s\treal: %s' % (self.sys, self.user, self.real)
124 | 
125 |   def print_raw(self):
126 |     print '%s\t%s\t%s' % (self.sys, self.user, self.real)
127 | 
128 |   def __eq__(self, other):
129 |     return self.sys == other.sys and \
130 |            self.user == other.user and \
131 |            self.real == other.real
132 | 
133 |   def __lt__(self, other):
134 |     if self.real != other.real:
135 |       return self.real < other.real
136 |     if self.user != other.user:
137 |       return self.user < other.user
138 |     if self.user != other.user:
139 |       return self.user < other.user
140 | 
141 | 
142 | 
143 | def add_result(regexp, engine, res):
144 |   if regexp not in results:
145 |     results[regexp] = {}
146 |   results[regexp][engine] = res
147 | 
148 | 
149 | 
150 | # Print the results. The list of results can be reduced via a function passed in
151 | # argument. The result must be iterable.
152 | def print_results(reduce_function = lambda x: x):
153 |   print 'regexp\tengine\tsys\tuser\treal'
154 |   # Benchmarks have been run in order, so keys have been added in reversed order
155 |   # to the dictionary of results.
156 |   for regexp in reversed(results.keys()):
157 |     for engine in sorted(results[regexp].keys()):
158 |       to_print = reduce_function(results[regexp][engine])
159 |       for res in to_print:
160 |         print '%s\t%s\t' % (regexp, engine),
161 |         res.print_raw()
162 | 
163 | 
164 | 
165 | def run_benchmarks():
166 |   # TODO: On OSX redirecting to /dev/null raises an error.
167 |   out_file = open('/tmp/gjrep.py.out', 'w')
168 |   for regexp in regexps:
169 |     for engine in engines:
170 |       options = ['']
171 |       if engine in engines_options_extra:
172 |         options = options + engines_options_extra[engine]
173 |       for opt in options:
174 |         results = []
175 | 
176 |         run_cmd = ['time', '-p', engine, opt] + engines_options_base[engine] + [quotify(regexp), args.directory, '>', '/tmp/gjrep.py.out']
177 |         print ' '.join(run_cmd)
178 | 
179 |         for i in range(1, args.iterations + 1):
180 |           p_cmd = subprocess.Popen(' '.join(run_cmd), stdout=out_file, stderr=subprocess.PIPE, shell=True)
181 |           p_g = subprocess.Popen(['grep', 'real\\|user\\|sys'], stdin=p_cmd.stderr,
182 |                                  stdout=subprocess.PIPE)
183 |           p_cmd.wait()
184 |           p_g.wait()
185 |           time_output = p_g.communicate()[0]
186 | 
187 |           results.append(TimeResult(time_output))
188 | 
189 |         engine_name = engine
190 |         if (len(opt) > 0):
191 |           engine_name = engine_name + ' ' + opt
192 |         add_result(regexp, ' '.join([engine, opt]), results)
193 |   out_file.close()
194 | 
195 | 
196 | 
197 | 
198 | run_benchmarks()
199 | print_results(lambda x : [min(x)])
200 | 


--------------------------------------------------------------------------------
/tools/benchmarks/resources/html/bench_plot.js:
--------------------------------------------------------------------------------
 1 | var kiB = 1024;
 2 | var MiB = kiB * kiB;
 3 | var GiB = MiB * kiB;
 4 | 
 5 | function suffix_format(val, suffix) {
 6 |   if (val > GiB)
 7 |     return (val / GiB).toFixed(1) + " G" + suffix;
 8 |   if (val > MiB)
 9 |     return (val / MiB).toFixed(0) + " M" + suffix;
10 |   else if (val > kiB)
11 |     return (val / kiB).toFixed(0) + " k" + suffix;
12 |   else
13 |     return (val / 1.0).toFixed(0) + " " + suffix;
14 | }
15 | 
16 | function byte_format(val) { return suffix_format(val, "iB"); }
17 | function byte_sec_format(val) { return suffix_format(val, "iB/s"); }
18 | 
19 | function yFormatter(val, axis) { return byte_sec_format(val); }
20 | 
21 | var plot_options_parallel = {
22 |   series: { lines: { show: true },
23 |             points: { show: true, radius: 2} },
24 |   legend: { position: "nw" },
25 |   grid:   { hoverable: true, clickable: true },
26 |   xaxis:  { ticks: [[8,"8 B"], [16,"16 B"], [64,"64 B"], [256,"256 B"], [kiB,"1 kiB"], [4*kiB,"4 kiB"], [16*kiB,"16 kiB"], [64*kiB,"64 kiB"], [256*kiB,"256 kiB"], [1*MiB,"1 MiB"], [4*MiB,"4 MiB"], [16*MiB,"16 MiB"]],
27 |             transform: function (v) { return Math.log(v); },
28 |             inverseTransform: function (v) { return Math.exp(v); } },
29 |   yaxis:  { tickFormatter: yFormatter }
30 | };
31 | 
32 | var plot_options_speed_time = {
33 |   series: { lines: { show: true },
34 |             points: { show: true, radius: 2} },
35 |   legend: { position: "nw" },
36 |   grid:   { hoverable: true, clickable: true },
37 |   xaxis:  { mode: "time" },
38 |   yaxis:  { tickFormatter: yFormatter }
39 | };
40 | 
41 | 
42 | function show_tooltip(x, y, color, contents) {
43 |   $('<div id="tooltip">' + contents + '</div>').css( {
44 |     position: 'absolute',
45 |     display: 'none',
46 |     top: y + 5,
47 |     left: x + 5,
48 |     border: '1px solid #fdd',
49 |     padding: '2px',
50 |     'background-color': color,
51 |     opacity: 0.80
52 |   }).appendTo("body").fadeIn(200);
53 | }
54 | 
55 | 
56 | function plothover_func(event, pos, item) {
57 |   $("#x").text(pos.x.toFixed(2));
58 |   $("#y").text(pos.y.toFixed(2));
59 | 
60 |   if (item) {
61 |     if (previousPoint != item.dataIndex) {
62 |       previousPoint = item.dataIndex;
63 | 
64 |       $("#tooltip").remove();
65 |       var x = item.datapoint[0].toFixed(2),
66 |           y = item.datapoint[1].toFixed(2);
67 | 
68 |       show_tooltip(item.pageX, item.pageY,
69 |                   item.series.color,
70 |                   item.series.label + '<br/>' +
71 |                   byte_sec_format(y) + ' (' + Math.floor(y) + ' B/s)' + '<br/>' +
72 |                   byte_format(x)  + ' (' + Math.floor(x) + ' B)');
73 |     }
74 |   }
75 |   else {
76 |     $("#tooltip").remove();
77 |     previousPoint = null;
78 |   }
79 | }
80 | 
81 | function plot_according_to_choices(graph_id, data_sets, choice_container) {
82 |   var data = [];
83 |   choice_container.find("input:checked").each(function () {
84 |     var key = $(this).attr("name");
85 |     if (key && data_sets[key])
86 |     data.push(data_sets[key]);
87 |   });
88 |   if (data.length > 0)
89 |     $.plot($("#" + graph_id), data, plot_options_parallel);
90 | }
91 | 


--------------------------------------------------------------------------------
/tools/benchmarks/resources/html/benchmarks_results.html.footer:
--------------------------------------------------------------------------------
1 | 
2 | 
3 |   </body>
4 | </html>
5 | 


--------------------------------------------------------------------------------
/tools/benchmarks/resources/html/benchmarks_results.html.header:
--------------------------------------------------------------------------------
 1 | <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
 2 | <!--
 3 | Copyright (C) 2013 Alexandre Rames <alexandre@coreperf.com>
 4 | 
 5 | This page is available under the Creative Commons "CC BY" licence.
 6 | -->
 7 | <html>
 8 |   <head>
 9 |     <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
10 |     <title>Rejit</title>
11 |     <link rel="stylesheet" href="resources/html/main.css">
12 |     <script language="javascript" type="text/javascript" src="resources/html/flot/jquery.js"></script>
13 |     <script language="javascript" type="text/javascript" src="resources/html/flot/jquery.flot.js"></script>
14 |     <script language="javascript" type="text/javascript" src="resources/html/flot/jquery.flot.time.js"></script>
15 |   </head>
16 |   <body>
17 |     <h1>Benchmarks</h1>
18 |     <p>
19 |     File automatically generated by Rejit benchmark tools.
20 |     See the <a href="http://www.coreperf.com/projects/rejit/documentation.html#rejit_benchmark_suite">documentation</a>
21 |     for instructions on how to run the benchmarks.
22 |     </p>
23 |     <p>
24 |     Benchmark results in this page present performance to search for all
25 |     (left-most longest) matches of a regular expression in a text.
26 |     <br />
27 |     Performance is reported in bytes per second: the size of the text matched
28 |     divided by the time to process it. This can be interpreted as speeds at
29 |     which files of given sizes can be processed.
30 |     </p>
31 | 
32 |     <script language="javascript" type="text/javascript" src="resources/html/bench_plot.js"></script>
33 | 
34 | 


--------------------------------------------------------------------------------
/tools/benchmarks/resources/html/main.css:
--------------------------------------------------------------------------------
 1 | 
 2 | /*******************************************************************************
 3 | Graphs plotting (using flot.js: http://www.flotcharts.org/)
 4 | *******************************************************************************/
 5 | 
 6 | .flot_choices {
 7 |   margin-left: 10px;
 8 | }
 9 | 
10 | .flot_choices li > div {
11 |   display: inline-block;
12 |   margin-right: 4px;
13 | }
14 | 
15 | .flot_choices li label {
16 |   margin-left: 3px;
17 | }
18 | 
19 | 


--------------------------------------------------------------------------------
/tools/benchmarks/resources/sample_bench_complex.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/coreperf/rejit/36855db0644d6f3dbad69dc65cf002c81510ea19/tools/benchmarks/resources/sample_bench_complex.png


--------------------------------------------------------------------------------
/tools/debug/lldb_rejit.py:
--------------------------------------------------------------------------------
  1 | import lldb
  2 | import operator
  3 | import re
  4 | 
  5 | commands_prefix = 'rejit_'
  6 | commands = []
  7 | 
  8 | 
  9 | class Command:
 10 |   def __init__(self, name, func, help):
 11 |     self.name = commands_prefix + name
 12 |     self.func = func
 13 |     self.help = help
 14 | 
 15 | 
 16 | 
 17 | # ----- Help -------------------------------------------------------------------
 18 | def fn_help(debugger, command, result, dict):
 19 |   print '''\
 20 | lldb helpers to debug rejit.
 21 | Availble commands:'''
 22 |   for c in commands:
 23 |     print '\t%s:' % c.name
 24 |     help = c.help
 25 |     help = re.sub('^', '\t\t', help)
 26 |     help = re.sub('\n', '\n\t\t', help)
 27 |     print help
 28 | help_help = 'Prints this message.'
 29 | help = Command('help', fn_help, help_help)
 30 | commands += [help]
 31 | 
 32 | 
 33 | 
 34 | # ----- State ------------------------------------------------------------------
 35 | kPointerSize = 8
 36 | 
 37 | def fn_state(debugger, command, result, dict):
 38 |   target = debugger.GetSelectedTarget()
 39 |   opts = lldb.SBExpressionOptions()
 40 |   def eval_expr(type, c_expr):
 41 |     return type(target.EvaluateExpression(c_expr, opts).GetValue())
 42 |   CalleeSavedRegsSize = eval_expr(int, 'CalleeSavedRegsSize()')
 43 |   kStateInfoSize = 5 * kPointerSize
 44 |   state_fp_relative_fields = [
 45 |     ('time_summary_last', -CalleeSavedRegsSize - kStateInfoSize),
 46 |     ('ff_position',       -CalleeSavedRegsSize - 1 * kPointerSize),
 47 |     ('ff_found_state',    -CalleeSavedRegsSize - 2 * kPointerSize),
 48 |     ('backward_match',    -CalleeSavedRegsSize - 3 * kPointerSize),
 49 |     ('forward_match',     -CalleeSavedRegsSize - 4 * kPointerSize),
 50 |     ('last_match_end',    -CalleeSavedRegsSize - 5 * kPointerSize),
 51 |   ]
 52 |   print 'raw stack:'
 53 |   debugger.HandleCommand('register read fp')
 54 |   debugger.HandleCommand('register read sp')
 55 |   debugger.HandleCommand('memory read $sp -c `($fp - $sp) / sizeof(void*) + 1` -f pointer')
 56 |   print ''
 57 |   print 'string start:\t',
 58 |   debugger.HandleCommand('register read r13')
 59 |   print 'string pointer:\t',
 60 |   debugger.HandleCommand('register read r14')
 61 |   print 'string end:\t',
 62 |   debugger.HandleCommand('register read r15')
 63 |   print ''
 64 |   print 'result matches:\t',
 65 |   debugger.HandleCommand('register read rbx')
 66 |   print 'ring index:\t',
 67 |   debugger.HandleCommand('register read r12')
 68 | 
 69 |   fp = eval_expr(int, '$rbp')
 70 |   sp = eval_expr(int, '$rsp')
 71 |   ring_size = (fp - sp - (kStateInfoSize + CalleeSavedRegsSize)) / kPointerSize
 72 |   if ring_size <= 0:
 73 |     # The stack has not been set up yet.
 74 |     return
 75 | 
 76 |   print ''
 77 |   print 'fields:'
 78 |   def read_pointer_at_fp_offset(fp_relative_field):
 79 |     print fp_relative_field[0] + '\t',
 80 |     debugger.HandleCommand('memory read `$fp + %d` -c 1 -f pointer' % fp_relative_field[1])
 81 |   for field in state_fp_relative_fields:
 82 |     read_pointer_at_fp_offset(field)
 83 |   print 'state ring:'
 84 |   debugger.HandleCommand('memory read $sp -c `%d` -f pointer' % ring_size)
 85 | 
 86 | help_state = '''\
 87 | Use from a rejit frame. Parses the stack and displays detailed information about
 88 | the current state.'''
 89 | state = Command('state', fn_state, help_state)
 90 | commands += [state]
 91 | 
 92 | 
 93 | 
 94 | 
 95 | 
 96 | 
 97 | commands = sorted(commands, key=operator.attrgetter('name'))
 98 | 
 99 | def install_command(c, debugger, dict):
100 |   debugger.HandleCommand(
101 |     'command script add -f lldb_rejit.%s %s' % (c.func.__name__, c.name))
102 | 
103 | 
104 | def __lldb_init_module (debugger, dict):
105 |   for c in commands:
106 |     install_command(c, debugger, dict)
107 |   print 'rejit debugging helpers have been installed. Use `%s` for help.' % help.name
108 | 


--------------------------------------------------------------------------------
/tools/tests/run.py:
--------------------------------------------------------------------------------
  1 | #!/usr/bin/python
  2 | 
  3 | # Copyright (C) 2013 Alexandre Rames <alexandre@coreperf.com>
  4 | 
  5 | # rejit is free software: you can redistribute it and/or modify
  6 | # it under the terms of the GNU General Public License as published by
  7 | # the Free Software Foundation, either version 3 of the License, or
  8 | # (at your option) any later version.
  9 | #
 10 | # This program is distributed in the hope that it will be useful,
 11 | # but WITHOUT ANY WARRANTY; without even the implied warranty of
 12 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 13 | # GNU General Public License for more details.
 14 | #
 15 | # You should have received a copy of the GNU General Public License
 16 | # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 17 | 
 18 | import sys
 19 | import os
 20 | from os.path import join, dirname
 21 | import argparse
 22 | import itertools
 23 | import subprocess
 24 | 
 25 | # Import rejit utils.
 26 | dir_tests = dirname(os.path.realpath(__file__))
 27 | dir_rejit = dir_tests
 28 | while 'SConstruct' not in os.listdir(dir_rejit):
 29 |   dir_rejit = os.path.realpath(join(dir_rejit, '..'))
 30 | sys.path.insert(0, join(dir_rejit, 'tools'))
 31 | import utils
 32 | 
 33 | 
 34 | 
 35 | script_description = \
 36 | '''
 37 | Run rejit tests.
 38 | The tests are run for a all possible combinations of options that are required
 39 | to be tested.
 40 | '''
 41 | 
 42 | 
 43 | 
 44 | def optionify(name):
 45 |   return '--' + name
 46 | 
 47 | 
 48 | 
 49 | # The options that can be tested are abstracted to provide an easy way to add
 50 | # new ones.
 51 | # Build options are options passed to scons, with a syntax like `scons opt=val`
 52 | # Run (short for runtime) options are options passed to the test program and
 53 | # control runtime flags in rejit.
 54 | # See the definition of `test_options` below.
 55 | 
 56 | # 'all' is a special value for the options. If specified, all other values of
 57 | # the option are tested.
 58 | class TestOption(object):
 59 |   def __init__(self, option_type, name, help,
 60 |                val_test_choices, val_test_default = None):
 61 |     self.name = name
 62 |     self.option_type = option_type
 63 |     self.help = help
 64 |     self.val_test_choices = val_test_choices
 65 |     if val_test_default is not None:
 66 |       self.val_test_default = val_test_default
 67 |     else:
 68 |       self.val_test_default = val_test_choices[0]
 69 | 
 70 |   type_build = 'build'
 71 |   type_run = 'run'
 72 | 
 73 |   def is_build_option(self):
 74 |     return self.option_type == TestOption.type_build
 75 |   def is_run_option(self):
 76 |     return self.option_type == TestOption.type_run
 77 | 
 78 | class BuildOption(TestOption):
 79 |   option_type = TestOption.type_build
 80 |   def __init__(self, name, help,
 81 |                val_test_choices, val_test_default = None):
 82 |     super(BuildOption, self).__init__(BuildOption.option_type, name, help, val_test_choices, val_test_default)
 83 |   def args_list(self, to_test):
 84 |     res = []
 85 |     if to_test == 'all':
 86 |       for val in self.val_test_choices:
 87 |         if val != 'all':
 88 |           res.append(self.name + '=' + val)
 89 |     else:
 90 |       res.append(self.name + '=' + to_test)
 91 |     return res
 92 | 
 93 | class RunOption(TestOption):
 94 |   option_type = TestOption.type_run
 95 |   def __init__(self, name, help,
 96 |                val_test_choices, val_test_default = None):
 97 |     super(RunOption, self).__init__(RunOption.option_type, name, help, val_test_choices, val_test_default)
 98 |   def args_list(self, to_test):
 99 |     res = []
100 |     if to_test == 'all':
101 |       for val in self.val_test_choices:
102 |         if val != 'all':
103 |           res.append(optionify(self.name) + '=' + val)
104 |     else:
105 |       res.append(optionify(self.name) + '=' + to_test)
106 |     return res
107 | 
108 | 
109 | 
110 | test_options = [
111 |   BuildOption('mode', 'Test with the specified build modes.',
112 |               val_test_choices=['all'] + utils.build_options_modes),
113 |   BuildOption('simd', 'Test with the specified SIMD configurations.',
114 |               val_test_choices=['all', 'on', 'off']),
115 |   RunOption('use_fast_forward', 'Test with the specified configurations for fast-forwarding.',
116 |             val_test_choices=['all', '1', '0']),
117 |   RunOption('use_ff_reduce', 'Test with the specified configurations for common substrings extraction.',
118 |             val_test_choices=['all', '1', '0']),
119 |   RunOption('use_parser_opt', 'Test with the specified configurations for parser level optimizations.',
120 |             val_test_choices=['all', '1', '0'])
121 | ]
122 | 
123 | test_build_options = [opt for opt in test_options if opt.is_build_option()]
124 | test_run_options = [opt for opt in test_options if opt.is_run_option()]
125 | 
126 | 
127 | 
128 | def ParserAddTestOptionsArguments(parser, test_opts):
129 |   for opt in test_opts:
130 |     parser.add_argument(optionify(opt.name),
131 |                         choices=opt.val_test_choices,
132 |                         default=opt.val_test_default,
133 |                         help=opt.help,
134 |                         action='store')
135 | 
136 | 
137 | parser = argparse.ArgumentParser(description=script_description,
138 |                                  formatter_class=argparse.ArgumentDefaultsHelpFormatter)
139 | parser.add_argument('-j', '--jobs', type=int, action='store',
140 |                     default=1,
141 |                     help='Number of jobs to run simultaneously for the *build* commands')
142 | ParserAddTestOptionsArguments(parser, test_options)
143 | args = parser.parse_args()
144 | 
145 | 
146 | # List build and run options applicable for this test run.
147 | test_build_args_list = map(lambda opt : opt.args_list(args.__dict__[opt.name]), test_build_options)
148 | test_run_args_list = map(lambda opt : opt.args_list(args.__dict__[opt.name]), test_run_options)
149 | # List all combinations of build and run options that will be tested.
150 | test_build_args_combinations = list(itertools.product(*test_build_args_list))
151 | test_run_args_combinations = list(itertools.product(*test_run_args_list))
152 | 
153 | 
154 | 
155 | options_format=''
156 | for opt in test_options:
157 |   options_format += optionify(opt.name) + '={:<' + str(max(map(len, opt.val_test_choices))) + '} '
158 | 
159 | 
160 | 
161 | for build_args in test_build_args_combinations:
162 |   # Build the test executable for the current build options.
163 |   scons_command = ["scons", "-C", dir_rejit, 'test-rejit',
164 |                    '-j', str(args.jobs), "benchtest=on", "modifiable_flags=on"]
165 |   scons_command += build_args
166 |   print ' '.join(scons_command)
167 |   p_scons = subprocess.Popen(scons_command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
168 |   scons_ret = p_scons.wait()
169 |   if scons_ret != 0:
170 |     print 'Command:'
171 |     print ' '.join(scons_command)
172 |     print 'Output:'
173 |     scons_output = p_scons.communicate()[0]
174 |     print scons_output
175 |     sys.exit(scons_ret)
176 | 
177 |   for run_args in test_run_args_combinations:
178 |     test_command = [join(utils.dir_build_latest, 'test-rejit')]
179 |     test_command += run_args
180 |     print ' '.join(test_command),
181 |     p_test = subprocess.Popen(test_command, stdout=subprocess.PIPE)
182 |     test_output = ''
183 |     test_ret = p_test.poll()
184 |     while test_ret is None:
185 |       test_output += p_test.communicate()[0]
186 |       test_ret = p_test.poll()
187 |     if test_ret != 0:
188 |       print 'FAILED'
189 |       print 'Output:'
190 |       print test_output
191 |     else:
192 |       print '\t\tsuccess'
193 | 


--------------------------------------------------------------------------------
/tools/utils.py:
--------------------------------------------------------------------------------
  1 | # Copyright (C) 2013 Alexandre Rames <alexandre@coreperf.com>
  2 | # rejit is free software: you can redistribute it and/or modify
  3 | # it under the terms of the GNU General Public License as published by
  4 | # the Free Software Foundation, either version 3 of the License, or
  5 | # (at your option) any later version.
  6 | #
  7 | # This program is distributed in the hope that it will be useful,
  8 | # but WITHOUT ANY WARRANTY; without even the implied warranty of
  9 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 10 | # GNU General Public License for more details.
 11 | #
 12 | # You should have received a copy of the GNU General Public License
 13 | # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 14 | 
 15 | import platform
 16 | import sys
 17 | import os
 18 | import subprocess
 19 | 
 20 | from os.path import join
 21 | 
 22 | 
 23 | def error_message(msg):
 24 |   print "ERROR: %s" % msg
 25 | 
 26 | def error(msg, rc = 1):
 27 |   error_message(msg)
 28 |   sys.exit(rc)
 29 | 
 30 | def check_rc(rc, message):
 31 |   if rc != 0:
 32 |     error(message, rc)
 33 | 
 34 | def warning(msg):
 35 |   print "WARNING: %s" % msg
 36 | 
 37 | 
 38 | # Path helpers -----------------------------------------------------------------
 39 | dir_tools = os.path.dirname(os.path.realpath(__file__))
 40 | dir_root  = os.path.abspath(join(dir_tools, '..'))
 41 | dir_rejit  = os.path.realpath(dir_root)
 42 | dir_src                    = join(dir_root, 'src')
 43 | dir_benchmarks             = join(dir_tools, 'benchmarks')
 44 | dir_benchmarks_engines     = join(dir_benchmarks, 'engines')
 45 | dir_benchmarks_resources   = join(dir_benchmarks, 'resources')
 46 | dir_benchmarks_resources_html   = join(dir_benchmarks_resources, 'html')
 47 | dir_build                  = join(dir_root, 'build')
 48 | dir_build_latest           = join(dir_root, 'build', 'latest')
 49 | 
 50 | 
 51 | def ensure_dir(path_name):
 52 |   if not os.path.exists(path_name):
 53 |     os.makedirs(path_name)
 54 | 
 55 | 
 56 | # Build helpers ----------------------------------------------------------------
 57 | build_options_modes = ['release', 'debug']
 58 | 
 59 | build_options_archs = ['x64']
 60 | def GuessArchitecture():
 61 |   id = platform.machine()
 62 |   if id == 'x86_64':
 63 |     return 'x64'
 64 |   else:
 65 |     return None
 66 | 
 67 | build_options_oses = ['macos', 'linux']
 68 | def GuessOS():
 69 |   id = platform.system()
 70 |   if id == 'Linux':
 71 |     return 'linux'
 72 |   elif id == 'Darwin':
 73 |     return 'macos'
 74 |   else:
 75 |     return None
 76 | 
 77 | def assert_available(command):
 78 |   p = subprocess.Popen(['which', command], stdout=subprocess.PIPE)
 79 |   rc = p.wait()
 80 |   if rc != 0:
 81 |     "Command not available: %s" % command
 82 |     sys.exit(rc)
 83 | 
 84 | # TODO: Find out how to pass on 'cwd' instead of 'cwd_'
 85 | def command_assert(command, failure_message = None, cwd_ = None):
 86 |   p = subprocess.Popen(command, stdout=subprocess.PIPE, cwd=cwd_)
 87 |   for line in p.stdout.readlines():
 88 |     print line,
 89 |   rc = p.wait()
 90 |   if rc != 0:
 91 |     if failure_message:
 92 |       error_message(failure_message)
 93 |     error("Failed command: %s" % ' '.join(command))
 94 | 
 95 | 
 96 | 
 97 | # Benchmarks helpers -----------------------------------------------------------
 98 | 
 99 | def rejit_commit():
100 |   return subprocess.check_output(['git', 'rev-parse', 'HEAD']).strip('\n')
101 | 
102 | def re2_commit():
103 |   return subprocess.check_output(['hg', 'id'], cwd=join(dir_benchmarks_engines, 're2', 'hg.re2')).strip('\n')
104 | 
105 | def pcre_commit():
106 |   p1 = subprocess.Popen(['svn', 'info'], cwd=join(dir_benchmarks_engines, 'pcre', 'svn.pcre'), stdout=subprocess.PIPE)
107 |   p2 = subprocess.Popen(['grep', '-i', 'revision'], stdout=subprocess.PIPE, stdin=p1.stdout)
108 |   p3 = subprocess.Popen(['awk', '{print $2}'], stdout=subprocess.PIPE, stdin=p2.stdout)
109 |   return p3.communicate()[0].rstrip(' \n\t')
110 | 
111 | def v8_commit():
112 |   return subprocess.check_output(['git', 'rev-parse', 'HEAD'], cwd=join(dir_benchmarks_engines, 'v8', 'git.v8')).strip('\n')
113 | 
114 | 
115 | 
116 | # Help messages ----------------------------------------------------------------
117 | 
118 | help_argp = \
119 | '''
120 | Could not find libargp.
121 | Please install libargp; this is the only extra step on OSX.
122 | You can install it via homebrew or macports, or manually install it from
123 | https://github.com/fizx/libbow-osx/tree/master/argp.
124 | If you install it outside of the standard lib and include path, please compile
125 | rejit as follow:
126 |   $ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:<path to libargp.a>
127 |   $ export CCFLAGS=$CCFLAGS:-I<path to argp.h>
128 |   $ scons benchmark
129 | '''
130 | 
131 | help_messages = {
132 |     'argp': help_argp
133 |     }
134 | 


--------------------------------------------------------------------------------