'id', COMPRESSION => '$COMPRESSION'},
36 | {NAME => 'name', COMPRESSION => '$COMPRESSION'}
37 |
38 | create '$TSDB_TABLE',
39 | {NAME => 't', VERSIONS => 1, COMPRESSION => '$COMPRESSION', BLOOMFILTER => '$BLOOMFILTER'}
40 | EOF
41 |
--------------------------------------------------------------------------------
/test/uid/TestNoSuchUniqueId.java:
--------------------------------------------------------------------------------
1 | // This file is part of OpenTSDB.
2 | // Copyright (C) 2010-2012 The OpenTSDB Authors.
3 | //
4 | // This program is free software: you can redistribute it and/or modify it
5 | // under the terms of the GNU Lesser General Public License as published by
6 | // the Free Software Foundation, either version 2.1 of the License, or (at your
7 | // option) any later version. This program is distributed in the hope that it
8 | // will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty
9 | // of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser
10 | // General Public License for more details. You should have received a copy
11 | // of the GNU Lesser General Public License along with this program. If not,
12 | // see
18 | * Implementations of this interface aren't expected to be synchronized.
19 | */
20 | public interface DataPoint {
21 |
22 | /**
23 | * Returns the timestamp (in seconds) associated with this data point.
24 | * @return A strictly positive, 32 bit integer.
25 | */
26 | long timestamp();
27 |
28 | /**
29 | * Tells whether or not the this data point is a value of integer type.
30 | * @return {@code true} if the {@code i}th value is of integer type,
31 | * {@code false} if it's of doubleing point type.
32 | */
33 | boolean isInteger();
34 |
35 | /**
36 | * Returns the value of the this data point as a {@code long}.
37 | * @throws ClassCastException if the {@code isInteger() == false}.
38 | */
39 | long longValue();
40 |
41 | /**
42 | * Returns the value of the this data point as a {@code double}.
43 | * @throws ClassCastException if the {@code isInteger() == true}.
44 | */
45 | double doubleValue();
46 |
47 | /**
48 | * Returns the value of the this data point as a {@code double}, even if
49 | * it's a {@code long}.
50 | * @return When {@code isInteger() == false}, this method returns the same
51 | * thing as {@link #doubleValue}. Otherwise, it returns the same thing as
52 | * {@link #longValue}'s return value casted to a {@code double}.
53 | */
54 | double toDouble();
55 |
56 | }
57 |
--------------------------------------------------------------------------------
/src/tsd/client/ValidatedTextBox.java:
--------------------------------------------------------------------------------
1 | // This file is part of OpenTSDB.
2 | // Copyright (C) 2010-2012 The OpenTSDB Authors.
3 | //
4 | // This program is free software: you can redistribute it and/or modify it
5 | // under the terms of the GNU Lesser General Public License as published by
6 | // the Free Software Foundation, either version 2.1 of the License, or (at your
7 | // option) any later version. This program is distributed in the hope that it
8 | // will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty
9 | // of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser
10 | // General Public License for more details. You should have received a copy
11 | // of the GNU Lesser General Public License along with this program. If not,
12 | // see " + uri + "");
44 | }
45 | // Cheap security check to avoid directory traversal attacks.
46 | // TODO(tsuna): This is certainly not sufficient.
47 | if (uri.indexOf("..", 3) > 0) {
48 | throw new BadRequestException("Malformed URI " + uri + "");
49 | }
50 | final int questionmark = uri.indexOf('?', 3);
51 | final int pathend = questionmark > 0 ? questionmark : uri.length();
52 | query.sendFile(staticroot + uri.substring(3, pathend),
53 | uri.contains("nocache") ? 0 : 31536000 /*=1yr*/);
54 | }
55 | }
56 |
--------------------------------------------------------------------------------
/opentsdb.spec.in:
--------------------------------------------------------------------------------
1 | # Put the RPM in the current directory.
2 | %define _rpmdir .
3 | # Find the tarball produced by `make dist' in the current directory.
4 | %define _sourcedir %(echo $PWD)
5 |
6 | Name: @PACKAGE@
7 | Version: @VERSION@
8 | Release: 1
9 | Summary: A scalable, distributed Time Series Database
10 | Packager: @PACKAGE_BUGREPORT@
11 | BuildArch: noarch
12 | Group: Service/Monitoring
13 | License: LGPLv2.1+
14 | URL: http://opentsdb.net
15 | Source: @PACKAGE@-@VERSION@.tar.gz
16 |
17 | Requires: gnuplot
18 |
19 | # Disable the stupid stuff rpm distros include in the build process by
20 | # default:
21 | # Disable any prep shell actions. replace them with simply 'true'
22 | %define __spec_prep_pre true
23 | %define __spec_prep_post true
24 | # Disable any build shell actions. replace them with simply 'true'
25 | %define __spec_build_pre cd %{_builddir}
26 | %define __spec_build_post true
27 | # Disable any install shell actions. replace them with simply 'true'
28 | %define __spec_install_pre cd %{_builddir}
29 | %define __spec_install_post true
30 | # Disable any clean shell actions. replace them with simply 'true'
31 | %define __spec_clean_pre cd %{_builddir}
32 | %define __spec_clean_post true
33 |
34 |
35 | %description
36 | OpenTSDB is a distributed, scalable Time Series Database (TSDB) written on top
37 | of HBase. OpenTSDB was written to address a common need: store, index and
38 | serve metrics collected from computer systems (network gear, operating
39 | systems, applications) at a large scale, and make this data easily accessible
40 | and graphable.
41 |
42 | Thanks to HBase's scalability, OpenTSDB allows you to collect many thousands
43 | of metrics from thousands of hosts and applications, at a high rate (every few
44 | seconds). OpenTSDB will never delete or downsample data and can easily store
45 | billions of data points.
46 |
47 | %prep
48 | %setup -q
49 |
50 |
51 | %build
52 | %configure
53 | make
54 |
55 | %install
56 | rm -rf %{buildroot}
57 | make install DESTDIR=%{buildroot}
58 | mkdir -p %{buildroot}/var/cache/opentsdb
59 |
60 |
61 | %clean
62 | rm -rf %{buildroot}
63 |
64 |
65 | %files
66 | %defattr(644,root,root,755)
67 | %attr(0755,root,root) %{_bindir}/*
68 | %attr(0755,root,root) %{_datarootdir}/opentsdb/*.sh
69 | %doc
70 | %{_datarootdir}/opentsdb
71 | %{_bindir}/tsdb
72 | %dir %{_localstatedir}/cache/opentsdb
73 |
74 |
75 | %changelog
76 |
77 |
--------------------------------------------------------------------------------
/NEWS:
--------------------------------------------------------------------------------
1 | OpenTSDB - User visible changes.
2 |
3 | * Version 1.1.1 (2013-??-??) [???????]
4 |
5 | Noteworthy changes:
6 | - UIDs are now assigned in a lock-less fashion.
7 |
8 |
9 | * Version 1.1.0 (2013-03-08) [12879d7]
10 |
11 | Noteworthy changes:
12 | - Licensing adjustment: allow LGPLv2.1+ in addition to LGPLv3+.
13 | - Various fixes used when customizing size of UniqueId. The default size
14 | is 3 bytes and is a compile-time constant rarely changed in practice.
15 | - New a new standard deviation aggregator, `dev'.
16 | - New `fgcolor', `bgcolor' and `smooth' query parameters to /q.
17 | - New `tz' query string parameter to allow specifying a custom time zone.
18 | - Stop accepting connections when shutting down.
19 | - A new `dropcaches' administrative command allows discarding in-memory
20 | caches. Right now these are UID mappings.
21 | - Browser history support in the web UI.
22 | - Allow "1d-ago" style input in date boxes.
23 | - Fix the 30d integer overflow in the web UI.
24 | - Add the ability to use mouse for drag-to-zoom on graphs.
25 | - Integration with Maven.
26 | - Work around a Netty performance bug, increasing write throughput by 10x.
27 | - Properly parse floating point values in scientific notations.
28 | - Allow tuning the number of worker threads or using OIO.
29 | - Fix auto-completion bug causing partial results to show in the web UI.
30 | - Various internal bug fixes.
31 | - Update all dependencies.
32 | - TSDB data compaction is now enabled by default.
33 |
34 |
35 | * Version 1.0.0 (2011-12-23) [66a6b42]
36 |
37 | Initial release:
38 | - Upload data points through a telnet-style protocol.
39 | - HTTP interface to query data in ASCII and PNG.
40 | - Efficient, fully asynchronous write path.
41 | - Synchronous / blocking read path (to be rewritten).
42 | - TSDB data compaction (disabled by default).
43 |
44 | -----
45 |
46 | Copyright (C) 2011-2012 The OpenTSDB Authors.
47 |
48 | This library is free software: you can redistribute it and/or modify it
49 | under the terms of the GNU Lesser General Public License as published
50 | by the Free Software Foundation, either version 2.1 of the License, or
51 | (at your option) any later version.
52 |
53 | This library is distributed in the hope that it will be useful,
54 | but WITHOUT ANY WARRANTY; without even the implied warranty of
55 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
56 | GNU Lesser General Public License for more details.
57 |
58 | You should have received a copy of the GNU Lesser General Public License
59 | along with this library. If not, see " % args[0]
69 | return 1
70 | global DRAINDIR
71 | port = int(args[1])
72 | DRAINDIR = args[2]
73 | if not os.path.isdir(DRAINDIR):
74 | os.makedirs(DRAINDIR)
75 | server = ThreadedTCPServer(("0.0.0.0", port), Handler)
76 | try:
77 | print "Use Ctrl-C to stop me."
78 | server.serve_forever()
79 | except KeyboardInterrupt:
80 | pass
81 |
82 |
83 | if __name__ == "__main__":
84 | sys.exit(main(sys.argv))
85 |
--------------------------------------------------------------------------------
/src/core/Aggregator.java:
--------------------------------------------------------------------------------
1 | // This file is part of OpenTSDB.
2 | // Copyright (C) 2010-2012 The OpenTSDB Authors.
3 | //
4 | // This program is free software: you can redistribute it and/or modify it
5 | // under the terms of the GNU Lesser General Public License as published by
6 | // the Free Software Foundation, either version 2.1 of the License, or (at your
7 | // option) any later version. This program is distributed in the hope that it
8 | // will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty
9 | // of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser
10 | // General Public License for more details. You should have received a copy
11 | // of the GNU Lesser General Public License along with this program. If not,
12 | // see
20 | * All aggregators must be stateless. All they can do is run through a
21 | * sequence of {@link Longs Longs} or {@link Doubles Doubles} and return an
22 | * aggregated value.
23 | */
24 | public interface Aggregator {
25 |
26 | /**
27 | * A sequence of {@code long}s.
28 | *
29 | * This interface is semantically equivalent to
30 | * {@code Iterator}.
31 | */
32 | public interface Longs {
33 |
34 | /**
35 | * Returns {@code true} if this sequence has more values.
36 | * {@code false} otherwise.
37 | */
38 | boolean hasNextValue();
39 |
40 | /**
41 | * Returns the next {@code long} value in this sequence.
42 | * @throws NoSuchElementException if calling {@link #hasNextValue} returns
43 | * {@code false}.
44 | */
45 | long nextLongValue();
46 |
47 | }
48 |
49 | /**
50 | * A sequence of {@code double}s.
51 | *
52 | * This interface is semantically equivalent to
53 | * {@code Iterator}.
54 | */
55 | public interface Doubles {
56 |
57 | /**
58 | * Returns {@code true} if this sequence has more values.
59 | * {@code false} otherwise.
60 | */
61 | boolean hasNextValue();
62 |
63 | /**
64 | * Returns the next {@code double} value in this sequence.
65 | * @throws NoSuchElementException if calling {@link #hasNextValue} returns
66 | * {@code false}.
67 | */
68 | double nextDoubleValue();
69 |
70 | }
71 |
72 | /**
73 | * Aggregates a sequence of {@code long}s.
74 | * @param values The sequence to aggregate.
75 | * @return The aggregated value.
76 | */
77 | long runLong(Longs values);
78 |
79 | /**
80 | * Aggregates a sequence of {@code double}s.
81 | * @param values The sequence to aggregate.
82 | * @return The aggregated value.
83 | */
84 | double runDouble(Doubles values);
85 |
86 | }
87 |
--------------------------------------------------------------------------------
/tsdb.in:
--------------------------------------------------------------------------------
1 | #!/bin/bash
2 |
3 | set -e
4 | me=`basename "$0"`
5 | mydir=`dirname "$0"`
6 | # Either:
7 | # abs_srcdir and abs_builddir are set: we're running in a dev tree
8 | # or pkgdatadir is set: we've been installed, we respect that.
9 | abs_srcdir='@abs_srcdir@'
10 | abs_builddir='@abs_builddir@'
11 | pkgdatadir='@pkgdatadir@'
12 | # Either we've been installed and pkgdatadir exists, or we haven't been
13 | # installed and abs_srcdir / abs_builddir aren't empty.
14 | test -d "$pkgdatadir" || test -n "$abs_srcdir$abs_builddir" || {
15 | echo >&2 "$me: Uh-oh, \`$pkgdatadir' doesn't exist, is OpenTSDB properly installed?"
16 | exit 1
17 | }
18 |
19 | if test -n "$pkgdatadir"; then
20 | localdir="$pkgdatadir"
21 | for jar in "$pkgdatadir"/*.jar; do
22 | CLASSPATH="$CLASSPATH:$jar"
23 | done
24 | # Add pkgdatadir itself so we can find logback.xml
25 | CLASSPATH="$CLASSPATH:$pkgdatadir"
26 | else
27 | localdir="$abs_builddir"
28 | # If we're running out of the build tree, it's especially important that we
29 | # know exactly what jars we need to build the CLASSPATH. Otherwise people
30 | # cannot easily pick up new dependencies as we might mix multiple versions
31 | # of the same dependencies on the CLASSPATH, which is bad. Looking for a
32 | # specific version of each jar prevents this problem.
33 | # TODO(tsuna): Once we jarjar all the dependencies together, this will no
34 | # longer be an issue. See issue #23.
35 | for jar in `make -C "$abs_builddir" printdeps | sed '/third_party.*jar/!d'`; do
36 | for dir in "$abs_builddir" "$abs_srcdir"; do
37 | test -f "$dir/$jar" && CLASSPATH="$CLASSPATH:$dir/$jar" && continue 2
38 | done
39 | echo >&2 "$me: error: Couldn't find \`$jar' either under \`$abs_builddir' or \`$abs_srcdir'."
40 | exit 2
41 | done
42 | # Add the src dir so we can find logback.xml
43 | CLASSPATH="$CLASSPATH:$abs_srcdir/src"
44 | fi
45 | # Remove any leading colon.
46 | CLASSPATH="${CLASSPATH#:}"
47 |
48 | usage() {
49 | echo >&2 "usage: $me [args]"
50 | echo 'Valid commands: fsck, import, mkmetric, query, tsd, scan, uid'
51 | exit 1
52 | }
53 |
54 | case $1 in
55 | (fsck)
56 | MAINCLASS=Fsck
57 | ;;
58 | (import)
59 | MAINCLASS=TextImporter
60 | ;;
61 | (mkmetric)
62 | shift
63 | set uid assign metrics "$@"
64 | MAINCLASS=UidManager
65 | ;;
66 | (query)
67 | MAINCLASS=CliQuery
68 | ;;
69 | (tsd)
70 | MAINCLASS=TSDMain
71 | ;;
72 | (scan)
73 | MAINCLASS=DumpSeries
74 | ;;
75 | (uid)
76 | MAINCLASS=UidManager
77 | ;;
78 | (*)
79 | echo >&2 "$me: error: unknown command '$1'"
80 | usage
81 | ;;
82 | esac
83 | shift
84 |
85 | JAVA=${JAVA-'java'}
86 | JVMARGS=${JVMARGS-'-enableassertions -enablesystemassertions'}
87 | test -r "$localdir/tsdb.local" && . "$localdir/tsdb.local"
88 | exec $JAVA $JVMARGS -classpath "$CLASSPATH" net.opentsdb.tools.$MAINCLASS "$@"
89 |
--------------------------------------------------------------------------------
/src/core/SeekableView.java:
--------------------------------------------------------------------------------
1 | // This file is part of OpenTSDB.
2 | // Copyright (C) 2010-2012 The OpenTSDB Authors.
3 | //
4 | // This program is free software: you can redistribute it and/or modify it
5 | // under the terms of the GNU Lesser General Public License as published by
6 | // the Free Software Foundation, either version 2.1 of the License, or (at your
7 | // option) any later version. This program is distributed in the hope that it
8 | // will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty
9 | // of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser
10 | // General Public License for more details. You should have received a copy
11 | // of the GNU Lesser General Public License along with this program. If not,
12 | // see zero-copy view to iterate through data points.
20 | *
21 | * The iterator returned by classes that implement this interface must return
22 | * each {@link DataPoint} in {@code O(1)} and does not support {@link #remove}.
23 | *
24 | * Because no data is copied during iteration and no new object gets created,
25 | * the {@link DataPoint} returned must not be stored and gets
26 | * invalidated as soon as {@link #next} is called on the iterator (actually it
27 | * doesn't get invalidated but rather its contents changes). If you want to
28 | * store individual data points, you need to copy the timestamp and value out
29 | * of each {@link DataPoint} into your own data structures.
30 | *
31 | * In the vast majority of cases, the iterator will be used to go once through
32 | * all the data points, which is why it's not a problem if the iterator acts
33 | * just as a transient "view". Iterating will be very cheap since no memory
34 | * allocation is required (except to instantiate the actual iterator at the
35 | * beginning).
36 | */
37 | public interface SeekableView extends Iterator {
38 |
39 | /**
40 | * Returns {@code true} if this view has more elements.
41 | */
42 | boolean hasNext();
43 |
44 | /**
45 | * Returns a view on the next data point.
46 | * No new object gets created, the referenced returned is always the same
47 | * and must not be stored since its internal data structure will change the
48 | * next time {@code next()} is called.
49 | * @throws NoSuchElementException if there were no more elements to iterate
50 | * on (in which case {@link #hasNext} would have returned {@code false}.
51 | */
52 | DataPoint next();
53 |
54 | /**
55 | * Unsupported operation.
56 | * @throws UnsupportedOperationException always.
57 | */
58 | void remove();
59 |
60 | /**
61 | * Advances the iterator to the given point in time.
62 | *
63 | * This allows the iterator to skip all the data points that are strictly
64 | * before the given timestamp.
65 | * @param timestamp A strictly positive 32 bit UNIX timestamp (in seconds).
66 | * @throws IllegalArgumentException if the timestamp is zero, or negative,
67 | * or doesn't fit on 32 bits (think "unsigned int" -- yay Java!).
68 | */
69 | void seek(long timestamp);
70 |
71 | }
72 |
--------------------------------------------------------------------------------
/src/tsd/client/QueryString.java:
--------------------------------------------------------------------------------
1 | // This file is part of OpenTSDB.
2 | // Copyright (C) 2012 The OpenTSDB Authors.
3 | //
4 | // This program is free software: you can redistribute it and/or modify it
5 | // under the terms of the GNU Lesser General Public License as published by
6 | // the Free Software Foundation, either version 2.1 of the License, or (at your
7 | // option) any later version. This program is distributed in the hope that it
8 | // will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty
9 | // of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser
10 | // General Public License for more details. You should have received a copy
11 | // of the GNU Lesser General Public License along with this program. If not,
12 | // see > {
25 |
26 | /**
27 | * Returns the decoded key-value parameter pairs of the URI.
28 | */
29 | public static QueryString decode(final String s) {
30 | final QueryString params = new QueryString();
31 | String name = null;
32 | int pos = 0; // Beginning of the unprocessed region
33 | int i; // End of the unprocessed region
34 | for (i = 0; i < s.length(); i++) {
35 | final char c = s.charAt(i);
36 | if (c == '=' && name == null) {
37 | if (pos != i) {
38 | name = s.substring(pos, i);
39 | }
40 | pos = i + 1;
41 | } else if (c == '&') {
42 | if (name == null && pos != i) {
43 | // We haven't seen an `=' so far but moved forward.
44 | // Must be a param of the form '&a&' so add it with
45 | // an empty value.
46 | params.add(s.substring(pos, i), "");
47 | } else if (name != null) {
48 | params.add(name, s.substring(pos, i));
49 | name = null;
50 | }
51 | pos = i + 1;
52 | }
53 | }
54 |
55 | if (pos != i) { // Are there characters we haven't dealt with?
56 | if (name == null) { // Yes and we haven't seen any `='.
57 | params.add(s.substring(pos, i), "");
58 | } else { // Yes and this must be the last value.
59 | params.add(name, s.substring(pos, i));
60 | }
61 | } else if (name != null) { // Have we seen a name without value?
62 | params.add(name, "");
63 | }
64 |
65 | return params;
66 | }
67 |
68 | /**
69 | * Adds a query string element.
70 | * @param name The name of the element.
71 | * @param value The value of the element.
72 | */
73 | public void add(final String name, final String value) {
74 | ArrayList values = super.get(name);
75 | if (values == null) {
76 | values = new ArrayList(1); // Often there's only 1 value.
77 | super.put(name, values);
78 | }
79 | values.add(value);
80 | }
81 |
82 | /**
83 | * Returns the first value for the given key, or {@code null}.
84 | */
85 | public String getFirst(final String key) {
86 | final ArrayList values = super.get(key);
87 | return values == null ? null : values.get(0);
88 | }
89 |
90 | }
91 |
--------------------------------------------------------------------------------
/src/tsd/client/EventsHandler.java:
--------------------------------------------------------------------------------
1 | // This file is part of OpenTSDB.
2 | // Copyright (C) 2010-2012 The OpenTSDB Authors.
3 | //
4 | // This program is free software: you can redistribute it and/or modify it
5 | // under the terms of the GNU Lesser General Public License as published by
6 | // the Free Software Foundation, either version 2.1 of the License, or (at your
7 | // option) any later version. This program is distributed in the hope that it
8 | // will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty
9 | // of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser
10 | // General Public License for more details. You should have received a copy
11 | // of the GNU Lesser General Public License along with this program. If not,
12 | // see
32 | * This handler is just a convenient 4-in-1 handler that can be re-used on a
33 | * wide range of widgets such as {@code TextBox} and its derivative,
34 | * {@code CheckBox}, {@code ListBox} etc.
35 | */
36 | abstract class EventsHandler implements BlurHandler, ChangeHandler,
37 | ClickHandler, KeyPressHandler {
38 |
39 | /**
40 | * Called after one of the events (click, blur, change or "enter"
41 | * is pressed) occurs.
42 | *
43 | * This method is NOT called while the event is happening. It's invoked via
44 | * a {@link DeferredCommand deferred command}. This entails that the event
45 | * can't be cancelled as it has already executed. The reason the call is
46 | * deferred is that this way, things like {@code SuggestBox}s will have
47 | * already done their auto-completion by the time this method is called, and
48 | * thus the handler will see the suggested text instead of the partial input
49 | * being typed by the user.
50 | * @param event The event that occurred.
51 | */
52 | protected abstract void onEvent(DomEvent event);
53 |
54 | public final void onClick(final ClickEvent event) {
55 | scheduleEvent(event);
56 | }
57 |
58 | public final void onBlur(final BlurEvent event) {
59 | scheduleEvent(event);
60 | }
61 |
62 | public final void onChange(final ChangeEvent event) {
63 | scheduleEvent(event);
64 | }
65 |
66 | public final void onKeyPress(final KeyPressEvent event) {
67 | if (event.getCharCode() == KeyCodes.KEY_ENTER) {
68 | scheduleEvent(event);
69 | }
70 | }
71 |
72 | /** Executes the event using a deferred command. */
73 | private void scheduleEvent(final DomEvent event) {
74 | DeferredCommand.addCommand(new Command() {
75 | public void execute() {
76 | onEvent(event);
77 | }
78 | });
79 | }
80 |
81 | }
82 |
--------------------------------------------------------------------------------
/src/uid/UniqueIdInterface.java:
--------------------------------------------------------------------------------
1 | // This file is part of OpenTSDB.
2 | // Copyright (C) 2010-2012 The OpenTSDB Authors.
3 | //
4 | // This program is free software: you can redistribute it and/or modify it
5 | // under the terms of the GNU Lesser General Public License as published by
6 | // the Free Software Foundation, either version 2.1 of the License, or (at your
7 | // option) any later version. This program is distributed in the hope that it
8 | // will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty
9 | // of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser
10 | // General Public License for more details. You should have received a copy
11 | // of the GNU Lesser General Public License along with this program. If not,
12 | // see
24 | * An instance of this class handles the unique IDs for one kind of ID. For
25 | * example:
26 | *
27 | * UniqueId metric_names = ...;
28 | * byte[] id = metric_names.get("sys.net.rx_bytes");
29 | *
30 | *
31 | * IDs are looked up in HBase and cached forever in memory (since they're
32 | * immutable). IDs are encoded on a fixed number of bytes, which is
33 | * implementation dependent.
34 | */
35 | public interface UniqueIdInterface {
36 |
37 | /**
38 | * Returns what kind of Unique ID is served by this instance.
39 | */
40 | String kind();
41 |
42 | /**
43 | * Returns the number of bytes on which each Unique ID is encoded.
44 | */
45 | short width();
46 |
47 | /**
48 | * Finds the name associated with a given ID.
49 | *
50 | * @param id The ID associated with that name.
51 | * @see #getId(String)
52 | * @see #getOrCreateId(String)
53 | * @throws NoSuchUniqueId if the given ID is not assigned.
54 | * @throws HBaseException if there is a problem communicating with HBase.
55 | * @throws IllegalArgumentException if the ID given in argument is encoded
56 | * on the wrong number of bytes.
57 | */
58 | String getName(byte[] id) throws NoSuchUniqueId, HBaseException;
59 |
60 | /**
61 | * Finds the ID associated with a given name.
62 | *
63 | * The length of the byte array is fixed in advance by the implementation.
64 | *
65 | * @param name The name to lookup in the table.
66 | * @see #getName(byte[])
67 | * @return A non-null, non-empty {@code byte[]} array.
68 | * @throws NoSuchUniqueName if the name requested doesn't have an ID assigned.
69 | * @throws HBaseException if there is a problem communicating with HBase.
70 | * @throws IllegalStateException if the ID found in HBase is encoded on the
71 | * wrong number of bytes.
72 | */
73 | byte[] getId(String name) throws NoSuchUniqueName, HBaseException;
74 |
75 | /**
76 | * Finds the ID associated with a given name or creates it.
77 | *
78 | * The length of the byte array is fixed in advance by the implementation.
79 | *
80 | * @param name The name to lookup in the table or to assign an ID to.
81 | * @throws HBaseException if there is a problem communicating with HBase.
82 | * @throws IllegalStateException if all possible IDs are already assigned.
83 | * @throws IllegalStateException if the ID found in HBase is encoded on the
84 | * wrong number of bytes.
85 | */
86 | byte[] getOrCreateId(String name) throws HBaseException, IllegalStateException;
87 |
88 | }
89 |
--------------------------------------------------------------------------------
/src/tsd/LineBasedFrameDecoder.java:
--------------------------------------------------------------------------------
1 | // This file is part of OpenTSDB.
2 | // Copyright (C) 2012 The OpenTSDB Authors.
3 | //
4 | // This program is free software: you can redistribute it and/or modify it
5 | // under the terms of the GNU Lesser General Public License as published by
6 | // the Free Software Foundation, either version 2.1 of the License, or (at your
7 | // option) any later version. This program is distributed in the hope that it
8 | // will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty
9 | // of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser
10 | // General Public License for more details. You should have received a copy
11 | // of the GNU Lesser General Public License along with this program. If not,
12 | // see
25 | * Both "\n" and "\r\n" are handled.
26 | *
27 | * This decoder is stateful and is thus NOT shareable.
28 | */
29 | final class LineBasedFrameDecoder extends FrameDecoder {
30 |
31 | /** Maximum length of a frame we're willing to decode. */
32 | private final int max_length;
33 | /** True if we're discarding input because we're already over max_length. */
34 | private boolean discarding;
35 |
36 | /**
37 | * Creates a new decoder.
38 | * @param max_length Maximum length of a frame we're willing to decode.
39 | * If a frame is longer than that, a {@link TooLongFrameException} will
40 | * be fired on the channel causing it.
41 | */
42 | public LineBasedFrameDecoder(final int max_length) {
43 | this.max_length = max_length;
44 | }
45 |
46 | @Override
47 | protected Object decode(final ChannelHandlerContext ctx,
48 | final Channel channel,
49 | final ChannelBuffer buffer) throws Exception {
50 | final int eol = findEndOfLine(buffer);
51 | if (eol != -1) {
52 | final ChannelBuffer frame;
53 | final int length = eol - buffer.readerIndex();
54 | assert length >= 0: "WTF? length=" + length;
55 | if (discarding) {
56 | frame = null;
57 | buffer.skipBytes(length);
58 | } else {
59 | frame = buffer.readBytes(length);
60 | }
61 | final byte delim = buffer.readByte();
62 | if (delim == '\r') {
63 | buffer.skipBytes(1); // Skip the \n.
64 | }
65 | return frame;
66 | }
67 |
68 | final int buffered = buffer.readableBytes();
69 | if (!discarding && buffered > max_length) {
70 | discarding = true;
71 | Channels.fireExceptionCaught(ctx.getChannel(),
72 | new TooLongFrameException("Frame length exceeds " + max_length + " ("
73 | + buffered + " bytes buffered already)"));
74 | }
75 | if (discarding) {
76 | buffer.skipBytes(buffer.readableBytes());
77 | }
78 | return null;
79 | }
80 |
81 | /**
82 | * Returns the index in the buffer of the end of line found.
83 | * Returns -1 if no end of line was found in the buffer.
84 | */
85 | private static int findEndOfLine(final ChannelBuffer buffer) {
86 | final int n = buffer.writerIndex();
87 | for (int i = buffer.readerIndex(); i < n; i ++) {
88 | final byte b = buffer.getByte(i);
89 | if (b == '\n') {
90 | return i;
91 | } else if (b == '\r' && i < n - 1 && buffer.getByte(i + 1) == '\n') {
92 | return i; // \r\n
93 | }
94 | }
95 | return -1; // Not found.
96 | }
97 |
98 | }
99 |
--------------------------------------------------------------------------------
/src/tsd/ConnectionManager.java:
--------------------------------------------------------------------------------
1 | // This file is part of OpenTSDB.
2 | // Copyright (C) 2010-2012 The OpenTSDB Authors.
3 | //
4 | // This program is free software: you can redistribute it and/or modify it
5 | // under the terms of the GNU Lesser General Public License as published by
6 | // the Free Software Foundation, either version 2.1 of the License, or (at your
7 | // option) any later version. This program is distributed in the hope that it
8 | // will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty
9 | // of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser
10 | // General Public License for more details. You should have received a copy
11 | // of the GNU Lesser General Public License along with this program. If not,
12 | // see
59 | * If {@link #setEndTime} was never called before, this method will
60 | * automatically execute
61 | * {@link #setEndTime setEndTime}{@code (System.currentTimeMillis() / 1000)}
62 | * to set the end time.
63 | * @return A strictly positive integer.
64 | */
65 | long getEndTime();
66 |
67 | /**
68 | * Sets the time series to the query.
69 | * @param metric The metric to retreive from the TSDB.
70 | * @param tags The set of tags of interest.
71 | * @param function The aggregation function to use.
72 | * @param rate If true, the rate of the series will be used instead of the
73 | * actual values.
74 | * @throws NoSuchUniqueName if the name of a metric, or a tag name/value
75 | * does not exist.
76 | */
77 | void setTimeSeries(String metric, Map tags,
78 | Aggregator function, boolean rate) throws NoSuchUniqueName;
79 |
80 | /**
81 | * Downsamples the results by specifying a fixed interval between points.
82 | *
83 | * Technically, downsampling means reducing the sampling interval. Here
84 | * the idea is similar. Instead of returning every single data point that
85 | * matched the query, we want one data point per fixed time interval. The
86 | * way we get this one data point is by aggregating all the data points of
87 | * that interval together using an {@link Aggregator}. This enables you
88 | * to compute things like the 5-minute average or 10 minute 99th percentile.
89 | * @param interval Number of seconds wanted between each data point.
90 | * @param downsampler Aggregation function to use to group data points
91 | * within an interval.
92 | */
93 | void downsample(int interval, Aggregator downsampler);
94 |
95 | /**
96 | * Runs this query.
97 | * @return The data points matched by this query.
98 | *
99 | * Each element in the non-{@code null} but possibly empty array returned
100 | * corresponds to one time series for which some data points have been
101 | * matched by the query.
102 | * @throws HBaseException if there was a problem communicating with HBase to
103 | * perform the search.
104 | */
105 | DataPoints[] run() throws HBaseException;
106 |
107 | }
108 |
--------------------------------------------------------------------------------
/src/core/DataPointsIterator.java:
--------------------------------------------------------------------------------
1 | // This file is part of OpenTSDB.
2 | // Copyright (C) 2010-2012 The OpenTSDB Authors.
3 | //
4 | // This program is free software: you can redistribute it and/or modify it
5 | // under the terms of the GNU Lesser General Public License as published by
6 | // the Free Software Foundation, either version 2.1 of the License, or (at your
7 | // option) any later version. This program is distributed in the hope that it
8 | // will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty
9 | // of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser
10 | // General Public License for more details. You should have received a copy
11 | // of the GNU Lesser General Public License along with this program. If not,
12 | // see ,
76 | Iterable {
77 |
78 | private final CyclicBufferAppender logbuf;
79 | private final StringBuilder buf = new StringBuilder(64);
80 | private int nevents;
81 |
82 | public LogIterator() {
83 | final Logger root =
84 | (Logger) LoggerFactory.getLogger(Logger.ROOT_LOGGER_NAME);
85 | logbuf = (CyclicBufferAppender) root.getAppender("CYCLIC");
86 | }
87 |
88 | public Iterator iterator() {
89 | nevents = logbuf.getLength();
90 | return this;
91 | }
92 |
93 | public boolean hasNext() {
94 | return nevents > 0;
95 | }
96 |
97 | public String next() {
98 | if (hasNext()) {
99 | nevents--;
100 | final ILoggingEvent event = (ILoggingEvent) logbuf.get(nevents);
101 | final String msg = event.getFormattedMessage();
102 | buf.setLength(0);
103 | buf.append(event.getTimeStamp() / 1000)
104 | .append('\t').append(event.getLevel().toString())
105 | .append('\t').append(event.getThreadName())
106 | .append('\t').append(event.getLoggerName())
107 | .append('\t').append(msg);
108 | final IThrowableProxy thrown = event.getThrowableProxy();
109 | if (thrown != null) {
110 | buf.append('\t').append(ThrowableProxyUtil.asString(thrown));
111 | }
112 | return buf.toString();
113 | }
114 | throw new NoSuchElementException("no more elements");
115 | }
116 |
117 | public void remove() {
118 | throw new UnsupportedOperationException();
119 | }
120 |
121 | }
122 |
123 | }
124 |
--------------------------------------------------------------------------------
/test/core/TestTags.java:
--------------------------------------------------------------------------------
1 | // This file is part of OpenTSDB.
2 | // Copyright (C) 2011-2012 The OpenTSDB Authors.
3 | //
4 | // This program is free software: you can redistribute it and/or modify it
5 | // under the terms of the GNU Lesser General Public License as published by
6 | // the Free Software Foundation, either version 2.1 of the License, or (at your
7 | // option) any later version. This program is distributed in the hope that it
8 | // will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty
9 | // of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser
10 | // General Public License for more details. You should have received a copy
11 | // of the GNU Lesser General Public License along with this program. If not,
12 | // see tags = new HashMap(2);
25 | Tags.parse(tags, "foo=bar");
26 | assertEquals(1, tags.size());
27 | assertEquals("bar", tags.get("foo"));
28 | Tags.parse(tags, "qux=baz");
29 | assertEquals(2, tags.size());
30 | assertEquals("bar", tags.get("foo"));
31 | assertEquals("baz", tags.get("qux"));
32 | }
33 |
34 | @Test(expected=IllegalArgumentException.class)
35 | public void parseNoEqualSign() {
36 | Tags.parse(new HashMap(1), "foo");
37 | }
38 |
39 | @Test(expected=IllegalArgumentException.class)
40 | public void parseTooManyEqualSigns() {
41 | Tags.parse(new HashMap(1), "foo=bar=qux");
42 | }
43 |
44 | @Test(expected=IllegalArgumentException.class)
45 | public void parseEmptyTagName() {
46 | Tags.parse(new HashMap(1), "=bar");
47 | }
48 |
49 | @Test(expected=IllegalArgumentException.class)
50 | public void parseEmptyTagValue() {
51 | Tags.parse(new HashMap(1), "foo=");
52 | }
53 |
54 | @Test(expected=IllegalArgumentException.class)
55 | public void parseDifferentValues() {
56 | final HashMap tags = new HashMap(1);
57 | Tags.parse(tags, "foo=bar");
58 | assertEquals(1, tags.size());
59 | assertEquals("bar", tags.get("foo"));
60 | Tags.parse(tags, "foo=qux");
61 | }
62 |
63 | @Test
64 | public void parseSameValues() {
65 | final HashMap tags = new HashMap(1);
66 | Tags.parse(tags, "foo=bar");
67 | assertEquals(1, tags.size());
68 | assertEquals("bar", tags.get("foo"));
69 | Tags.parse(tags, "foo=bar");
70 | assertEquals(1, tags.size());
71 | assertEquals("bar", tags.get("foo"));
72 | }
73 |
74 | @Test
75 | public void validateGoodString() {
76 | Tags.validateString("test", "omg-TSDB/42._foo_");
77 | }
78 |
79 | @Test(expected=IllegalArgumentException.class)
80 | public void validateNullString() {
81 | Tags.validateString("test", null);
82 | }
83 |
84 | @Test(expected=IllegalArgumentException.class)
85 | public void validateBadString() {
86 | Tags.validateString("test", "this is a test!");
87 | }
88 |
89 | // parseLong
90 |
91 | @Test
92 | public void parseLongSimple() {
93 | assertEquals(0, Tags.parseLong("0"));
94 | assertEquals(0, Tags.parseLong("+0"));
95 | assertEquals(0, Tags.parseLong("-0"));
96 | assertEquals(1, Tags.parseLong("1"));
97 | assertEquals(1, Tags.parseLong("+1"));
98 | assertEquals(-1, Tags.parseLong("-1"));
99 | assertEquals(4242, Tags.parseLong("4242"));
100 | assertEquals(4242, Tags.parseLong("+4242"));
101 | assertEquals(-4242, Tags.parseLong("-4242"));
102 | }
103 |
104 | @Test
105 | public void parseLongMaxValue() {
106 | assertEquals(Long.MAX_VALUE, Tags.parseLong(Long.toString(Long.MAX_VALUE)));
107 | }
108 |
109 | @Test
110 | public void parseLongMinValue() {
111 | assertEquals(Long.MIN_VALUE, Tags.parseLong(Long.toString(Long.MIN_VALUE)));
112 | }
113 |
114 | @Test(expected=NumberFormatException.class)
115 | public void parseLongEmptyString() {
116 | Tags.parseLong("");
117 | }
118 |
119 | @Test(expected=NumberFormatException.class)
120 | public void parseLongMalformed() {
121 | Tags.parseLong("42a51");
122 | }
123 |
124 | @Test(expected=NumberFormatException.class)
125 | public void parseLongMalformedPlus() {
126 | Tags.parseLong("+");
127 | }
128 |
129 | @Test(expected=NumberFormatException.class)
130 | public void parseLongMalformedMinus() {
131 | Tags.parseLong("-");
132 | }
133 |
134 | @Test(expected=NumberFormatException.class)
135 | public void parseLongValueTooLarge() {
136 | Tags.parseLong("18446744073709551616");
137 | }
138 |
139 | @Test(expected=NumberFormatException.class)
140 | public void parseLongValueTooLargeSubtle() {
141 | Tags.parseLong("9223372036854775808"); // MAX_VALUE + 1
142 | }
143 |
144 | @Test(expected=NumberFormatException.class)
145 | public void parseLongValueTooSmallSubtle() {
146 | Tags.parseLong("-9223372036854775809"); // MIN_VALUE - 1
147 | }
148 |
149 | }
150 |
--------------------------------------------------------------------------------
/src/core/Internal.java:
--------------------------------------------------------------------------------
1 | // This file is part of OpenTSDB.
2 | // Copyright (C) 2011-2012 The OpenTSDB Authors.
3 | //
4 | // This program is free software: you can redistribute it and/or modify it
5 | // under the terms of the GNU Lesser General Public License as published by
6 | // the Free Software Foundation, either version 2.1 of the License, or (at your
7 | // option) any later version. This program is distributed in the hope that it
8 | // will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty
9 | // of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser
10 | // General Public License for more details. You should have received a copy
11 | // of the GNU Lesser General Public License along with this program. If not,
12 | // see This class is not part of the public API.
24 | *
25 | * ,____________________________,
26 | * | This class is reserved for |
27 | * | OpenTSDB's internal usage! |
28 | * `----------------------------'
29 | * \ / \ //\
30 | * \ |\___/| / \// \\
31 | * /0 0 \__ / // | \ \
32 | * / / \/_/ // | \ \
33 | * @_^_@'/ \/_ // | \ \
34 | * //_^_/ \/_ // | \ \
35 | * ( //) | \/// | \ \
36 | * ( / /) _|_ / ) // | \ _\
37 | * ( // /) '/,_ _ _/ ( ; -. | _ _\.-~ .-~~~^-.
38 | * (( / / )) ,-{ _ `-.|.-~-. .~ `.
39 | * (( // / )) '/\ / ~-. _ .-~ .-~^-. \
40 | * (( /// )) `. { } / \ \
41 | * (( / )) .----~-.\ \-' .~ \ `. \^-.
42 | * ///.----../ \ _ -~ `. ^-` ^-_
43 | * ///-._ _ _ _ _ _ _}^ - - - - ~ ~-- ,.-~
44 | * /.-~
45 | * You've been warned by the dragon!
46 | *
47 | * This class is reserved for OpenTSDB's own internal usage only. If you use
48 | * anything from this package outside of OpenTSDB, a dragon will spontaneously
49 | * appear and eat you. You've been warned.
50 | *
51 | * This class only exists because Java's packaging system is annoying as the
52 | * "package-private" accessibility level only applies to the current package
53 | * but not its sub-packages, and because Java doesn't have fine-grained API
54 | * visibility mechanism such as that of Scala or C++.
55 | *
56 | * This package provides access into internal methods for higher-level
57 | * packages, for the sake of reducing code duplication and (ab)use of
58 | * reflection.
59 | */
60 | public final class Internal {
61 |
62 | /** @see Const#FLAG_BITS */
63 | public static final short FLAG_BITS = Const.FLAG_BITS;
64 |
65 | /** @see Const#LENGTH_MASK */
66 | public static final short LENGTH_MASK = Const.LENGTH_MASK;
67 |
68 | /** @see Const#FLAGS_MASK */
69 | public static final short FLAGS_MASK = Const.FLAGS_MASK;
70 |
71 | private Internal() {
72 | // Can't instantiate.
73 | }
74 |
75 | /** @see TsdbQuery#getScanner */
76 | public static Scanner getScanner(final Query query) {
77 | return ((TsdbQuery) query).getScanner();
78 | }
79 |
80 | /** @see RowKey#metricName */
81 | public static String metricName(final TSDB tsdb, final byte[] id) {
82 | return RowKey.metricName(tsdb, id);
83 | }
84 |
85 | /** Extracts the timestamp from a row key. */
86 | public static long baseTime(final TSDB tsdb, final byte[] row) {
87 | return Bytes.getUnsignedInt(row, tsdb.metrics.width());
88 | }
89 |
90 | /** @see Tags#getTags */
91 | public static Map getTags(final TSDB tsdb, final byte[] row) {
92 | return Tags.getTags(tsdb, row);
93 | }
94 |
95 | /** @see RowSeq#extractIntegerValue */
96 | public static long extractIntegerValue(final byte[] values,
97 | final int value_idx,
98 | final byte flags) {
99 | return RowSeq.extractIntegerValue(values, value_idx, flags);
100 | }
101 |
102 | /** @see RowSeq#extractFloatingPointValue */
103 | public static double extractFloatingPointValue(final byte[] values,
104 | final int value_idx,
105 | final byte flags) {
106 | return RowSeq.extractFloatingPointValue(values, value_idx, flags);
107 | }
108 |
109 | public static short metricWidth(final TSDB tsdb) {
110 | return tsdb.metrics.width();
111 | }
112 |
113 | /** @see CompactionQueue#complexCompact */
114 | public static KeyValue complexCompact(final KeyValue kv) {
115 | final ArrayList kvs = new ArrayList(1);
116 | kvs.add(kv);
117 | return CompactionQueue.complexCompact(kvs, kv.qualifier().length / 2);
118 | }
119 |
120 | }
121 |
--------------------------------------------------------------------------------
/src/tsd/PutDataPointRpc.java:
--------------------------------------------------------------------------------
1 | // This file is part of OpenTSDB.
2 | // Copyright (C) 2010-2012 The OpenTSDB Authors.
3 | //
4 | // This program is free software: you can redistribute it and/or modify it
5 | // under the terms of the GNU Lesser General Public License as published by
6 | // the Free Software Foundation, either version 2.1 of the License, or (at your
7 | // option) any later version. This program is distributed in the hope that it
8 | // will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty
9 | // of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser
10 | // General Public License for more details. You should have received a copy
11 | // of the GNU Lesser General Public License along with this program. If not,
12 | // see execute(final TSDB tsdb, final Channel chan,
38 | final String[] cmd) {
39 | requests.incrementAndGet();
40 | String errmsg = null;
41 | try {
42 | final class PutErrback implements Callback {
43 | public Exception call(final Exception arg) {
44 | if (chan.isConnected()) {
45 | chan.write("put: HBase error: " + arg.getMessage() + '\n');
46 | }
47 | hbase_errors.incrementAndGet();
48 | return arg;
49 | }
50 | public String toString() {
51 | return "report error to channel";
52 | }
53 | }
54 | return importDataPoint(tsdb, cmd).addErrback(new PutErrback());
55 | } catch (NumberFormatException x) {
56 | errmsg = "put: invalid value: " + x.getMessage() + '\n';
57 | invalid_values.incrementAndGet();
58 | } catch (IllegalArgumentException x) {
59 | errmsg = "put: illegal argument: " + x.getMessage() + '\n';
60 | illegal_arguments.incrementAndGet();
61 | } catch (NoSuchUniqueName x) {
62 | errmsg = "put: unknown metric: " + x.getMessage() + '\n';
63 | unknown_metrics.incrementAndGet();
64 | }
65 | if (errmsg != null && chan.isConnected()) {
66 | chan.write(errmsg);
67 | }
68 | return Deferred.fromResult(null);
69 | }
70 |
71 | /**
72 | * Collects the stats and metrics tracked by this instance.
73 | * @param collector The collector to use.
74 | */
75 | public static void collectStats(final StatsCollector collector) {
76 | collector.record("rpc.received", requests, "type=put");
77 | collector.record("rpc.errors", hbase_errors, "type=hbase_errors");
78 | collector.record("rpc.errors", invalid_values, "type=invalid_values");
79 | collector.record("rpc.errors", illegal_arguments, "type=illegal_arguments");
80 | collector.record("rpc.errors", unknown_metrics, "type=unknown_metrics");
81 | }
82 |
83 | /**
84 | * Imports a single data point.
85 | * @param tsdb The TSDB to import the data point into.
86 | * @param words The words describing the data point to import, in
87 | * the following format: {@code [metric, timestamp, value, ..tags..]}
88 | * @return A deferred object that indicates the completion of the request.
89 | * @throws NumberFormatException if the timestamp or value is invalid.
90 | * @throws IllegalArgumentException if any other argument is invalid.
91 | * @throws NoSuchUniqueName if the metric isn't registered.
92 | */
93 | private Deferred importDataPoint(final TSDB tsdb, final String[] words) {
94 | words[0] = null; // Ditch the "put".
95 | if (words.length < 5) { // Need at least: metric timestamp value tag
96 | // ^ 5 and not 4 because words[0] is "put".
97 | throw new IllegalArgumentException("not enough arguments"
98 | + " (need least 4, got " + (words.length - 1) + ')');
99 | }
100 | final String metric = words[1];
101 | if (metric.length() <= 0) {
102 | throw new IllegalArgumentException("empty metric name");
103 | }
104 | final long timestamp = Tags.parseLong(words[2]);
105 | if (timestamp <= 0) {
106 | throw new IllegalArgumentException("invalid timestamp: " + timestamp);
107 | }
108 | final String value = words[3];
109 | if (value.length() <= 0) {
110 | throw new IllegalArgumentException("empty value");
111 | }
112 | final HashMap tags = new HashMap();
113 | for (int i = 4; i < words.length; i++) {
114 | if (!words[i].isEmpty()) {
115 | Tags.parse(tags, words[i]);
116 | }
117 | }
118 | if (Tags.looksLikeInteger(value)) {
119 | return tsdb.addPoint(metric, timestamp, Tags.parseLong(value), tags);
120 | } else { // floating point value
121 | return tsdb.addPoint(metric, timestamp, Float.parseFloat(value), tags);
122 | }
123 | }
124 | }
125 |
--------------------------------------------------------------------------------
/src/core/WritableDataPoints.java:
--------------------------------------------------------------------------------
1 | // This file is part of OpenTSDB.
2 | // Copyright (C) 2010-2012 The OpenTSDB Authors.
3 | //
4 | // This program is free software: you can redistribute it and/or modify it
5 | // under the terms of the GNU Lesser General Public License as published by
6 | // the Free Software Foundation, either version 2.1 of the License, or (at your
7 | // option) any later version. This program is distributed in the hope that it
8 | // will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty
9 | // of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser
10 | // General Public License for more details. You should have received a copy
11 | // of the GNU Lesser General Public License along with this program. If not,
12 | // see
24 | * Implementations of this interface aren't expected to be synchronized.
25 | */
26 | public interface WritableDataPoints extends DataPoints {
27 |
28 | /**
29 | * Sets the metric name and tags of the series.
30 | *
31 | * This method can be called multiple times on the same instance to start
32 | * adding data points to another time series without having to create a new
33 | * instance.
34 | * @param metric A non-empty string.
35 | * @param tags The tags on this series. This map must be non-empty.
36 | * @throws IllegalArgumentException if the metric name is empty or contains
37 | * illegal characters.
38 | * @throws IllegalArgumentException if the tags list is empty or one of the
39 | * elements contains illegal characters.
40 | */
41 | void setSeries(String metric, Map tags);
42 |
43 | /**
44 | * Adds a {@code long} data point to the TSDB.
45 | *
46 | * The data point is immediately persisted unless {@link #setBufferingTime}
47 | * is used. Data points must be added in chronological order.
48 | * @param timestamp The timestamp associated with the value.
49 | * @param value The value of the data point.
50 | * @return A deferred object that indicates the completion of the request.
51 | * The {@link Object} has not special meaning and can be {@code null} (think
52 | * of it as {@code Deferred}). But you probably want to attach at
53 | * least an errback to this {@code Deferred} to handle failures.
54 | * @throws IllegalArgumentException if the timestamp is less than or equal
55 | * to the previous timestamp added or 0 for the first timestamp, or if the
56 | * difference with the previous timestamp is too large.
57 | * @throws HBaseException (deferred) if there was a problem while persisting
58 | * data.
59 | */
60 | Deferred addPoint(long timestamp, long value);
61 |
62 | /**
63 | * Appends a {@code float} data point to this sequence.
64 | *
65 | * The data point is immediately persisted unless {@link #setBufferingTime}
66 | * is used. Data points must be added in chronological order.
67 | * @param timestamp The timestamp associated with the value.
68 | * @param value The value of the data point.
69 | * @return A deferred object that indicates the completion of the request.
70 | * The {@link Object} has not special meaning and can be {@code null} (think
71 | * of it as {@code Deferred}). But you probably want to attach at
72 | * least an errback to this {@code Deferred} to handle failures.
73 | * @throws IllegalArgumentException if the timestamp is less than or equal
74 | * to the previous timestamp added or 0 for the first timestamp, or if the
75 | * difference with the previous timestamp is too large.
76 | * @throws IllegalArgumentException if the value is {@code NaN} or
77 | * {@code Infinite}.
78 | * @throws HBaseException (deferred) if there was a problem while persisting
79 | * data.
80 | */
81 | Deferred addPoint(long timestamp, float value);
82 |
83 | /**
84 | * Specifies for how long to buffer edits, in milliseconds.
85 | *
86 | * By calling this method, you're allowing new data points to be buffered
87 | * before being sent to HBase. {@code 0} (the default) means data points
88 | * are persisted immediately.
89 | *
90 | * Buffering improves performance, reduces the number of RPCs sent to HBase,
91 | * but can cause data loss if we die before we get a chance to send buffered
92 | * edits to HBase. It also entails that buffered data points aren't visible
93 | * to other applications using the TSDB until they're flushed to HBase.
94 | * @param time The approximate maximum number of milliseconds for which data
95 | * points should be buffered before being sent to HBase. This deadline will
96 | * be honored on a "best effort" basis.
97 | */
98 | void setBufferingTime(short time);
99 |
100 | /**
101 | * Specifies whether or not this is a batch import.
102 | *
103 | * It is preferred that this method be called for anything importing a batch
104 | * of data points (as opposed to streaming in new data points in real time).
105 | *
106 | * Calling this method changes a few important things:
107 | *
108 | * Data points may not be persisted immediately. In the event of an
109 | * outage in HBase during or slightly after the import, un-persisted data
110 | * points will be lost.
111 | * {@link #setBufferingTime} may be called with an argument
112 | * chosen by the implementation.
113 | *
114 | * @param batchornot if true, then this is a batch import.
115 | */
116 | void setBatchImport(boolean batchornot);
117 |
118 | }
119 |
--------------------------------------------------------------------------------
/test/stats/TestHistogram.java:
--------------------------------------------------------------------------------
1 | // This file is part of OpenTSDB.
2 | // Copyright (C) 2010-2012 The OpenTSDB Authors.
3 | //
4 | // This program is free software: you can redistribute it and/or modify it
5 | // under the terms of the GNU Lesser General Public License as published by
6 | // the Free Software Foundation, either version 2.1 of the License, or (at your
7 | // option) any later version. This program is distributed in the hope that it
8 | // will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty
9 | // of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser
10 | // General Public License for more details. You should have received a copy
11 | // of the GNU Lesser General Public License along with this program. If not,
12 | // see
21 | * Implementations of this interface aren't expected to be synchronized.
22 | */
23 | public interface DataPoints extends Iterable {
24 |
25 | /**
26 | * Returns the name of the series.
27 | */
28 | String metricName();
29 |
30 | /**
31 | * Returns the tags associated with these data points.
32 | * @return A non-{@code null} map of tag names (keys), tag values (values).
33 | */
34 | Map getTags();
35 |
36 | /**
37 | * Returns the tags associated with some but not all of the data points.
38 | *
39 | * When this instance represents the aggregation of multiple time series
40 | * (same metric but different tags), {@link #getTags} returns the tags that
41 | * are common to all data points (intersection set) whereas this method
42 | * returns all the tags names that are not common to all data points (union
43 | * set minus the intersection set, also called the symmetric difference).
44 | *
45 | * If this instance does not represent an aggregation of multiple time
46 | * series, the list returned is empty.
47 | * @return A non-{@code null} list of tag names.
48 | */
49 | List getAggregatedTags();
50 |
51 | /**
52 | * Returns the number of data points.
53 | *
54 | * This method must be implemented in {@code O(1)} or {@code O(n)}
55 | * where n = {@link #aggregatedSize} > 0.
56 | * @return A positive integer.
57 | */
58 | int size();
59 |
60 | /**
61 | * Returns the number of data points aggregated in this instance.
62 | *
63 | * When this instance represents the aggregation of multiple time series
64 | * (same metric but different tags), {@link #size} returns the number of data
65 | * points after aggregation, whereas this method returns the number of data
66 | * points before aggregation.
67 | *
68 | * If this instance does not represent an aggregation of multiple time
69 | * series, then 0 is returned.
70 | * @return A positive integer.
71 | */
72 | int aggregatedSize();
73 |
74 | /**
75 | * Returns a zero-copy view to go through {@code size()} data points.
76 | *
77 | * The iterator returned must return each {@link DataPoint} in {@code O(1)}.
78 | * The {@link DataPoint} returned must not be stored and gets
79 | * invalidated as soon as {@code next} is called on the iterator. If you
80 | * want to store individual data points, you need to copy the timestamp
81 | * and value out of each {@link DataPoint} into your own data structures.
82 | */
83 | SeekableView iterator();
84 |
85 | /**
86 | * Returns the timestamp associated with the {@code i}th data point.
87 | * The first data point has index 0.
88 | *
89 | * This method must be implemented in
90 | * O({@link #aggregatedSize}) or better.
91 | *
92 | * It is guaranteed that
timestamp(i) < timestamp(i+1)
93 | * @return A strictly positive integer.
94 | * @throws IndexOutOfBoundsException if {@code i} is not in the range
95 | * [0, {@link #size} - 1]
96 | */
97 | long timestamp(int i);
98 |
99 | /**
100 | * Tells whether or not the {@code i}th value is of integer type.
101 | * The first data point has index 0.
102 | *
103 | * This method must be implemented in
104 | * O({@link #aggregatedSize}) or better.
105 | * @return {@code true} if the {@code i}th value is of integer type,
106 | * {@code false} if it's of floating point type.
107 | * @throws IndexOutOfBoundsException if {@code i} is not in the range
108 | * [0, {@link #size} - 1]
109 | */
110 | boolean isInteger(int i);
111 |
112 | /**
113 | * Returns the value of the {@code i}th data point as a long.
114 | * The first data point has index 0.
115 | *
116 | * This method must be implemented in
117 | * O({@link #aggregatedSize}) or better.
118 | * Use {@link #iterator} to get successive {@code O(1)} accesses.
119 | * @see #iterator
120 | * @throws IndexOutOfBoundsException if {@code i} is not in the range
121 | * [0, {@link #size} - 1]
122 | * @throws ClassCastException if the
123 | * {@link #isInteger isInteger(i)} == false.
124 | */
125 | long longValue(int i);
126 |
127 | /**
128 | * Returns the value of the {@code i}th data point as a float.
129 | * The first data point has index 0.
130 | *
131 | * This method must be implemented in
132 | * O({@link #aggregatedSize}) or better.
133 | * Use {@link #iterator} to get successive {@code O(1)} accesses.
134 | * @see #iterator
135 | * @throws IndexOutOfBoundsException if {@code i} is not in the range
136 | * [0, {@link #size} - 1]
137 | * @throws ClassCastException if the
138 | * {@link #isInteger isInteger(i)} == true.
139 | */
140 | double doubleValue(int i);
141 |
142 | }
143 |
--------------------------------------------------------------------------------
/src/core/Aggregators.java:
--------------------------------------------------------------------------------
1 | // This file is part of OpenTSDB.
2 | // Copyright (C) 2010-2012 The OpenTSDB Authors.
3 | //
4 | // This program is free software: you can redistribute it and/or modify it
5 | // under the terms of the GNU Lesser General Public License as published by
6 | // the Free Software Foundation, either version 2.1 of the License, or (at your
7 | // option) any later version. This program is distributed in the hope that it
8 | // will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty
9 | // of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser
10 | // General Public License for more details. You should have received a copy
11 | // of the GNU Lesser General Public License along with this program. If not,
12 | // see aggregators;
41 |
42 | static {
43 | aggregators = new HashMap(5);
44 | aggregators.put("sum", SUM);
45 | aggregators.put("min", MIN);
46 | aggregators.put("max", MAX);
47 | aggregators.put("avg", AVG);
48 | aggregators.put("dev", DEV);
49 | }
50 |
51 | private Aggregators() {
52 | // Can't create instances of this utility class.
53 | }
54 |
55 | /**
56 | * Returns the set of the names that can be used with {@link #get get}.
57 | */
58 | public static Set set() {
59 | return aggregators.keySet();
60 | }
61 |
62 | /**
63 | * Returns the aggregator corresponding to the given name.
64 | * @param name The name of the aggregator to get.
65 | * @throws NoSuchElementException if the given name doesn't exist.
66 | * @see #set
67 | */
68 | public static Aggregator get(final String name) {
69 | final Aggregator agg = aggregators.get(name);
70 | if (agg != null) {
71 | return agg;
72 | }
73 | throw new NoSuchElementException("No such aggregator: " + name);
74 | }
75 |
76 | private static final class Sum implements Aggregator {
77 |
78 | public long runLong(final Longs values) {
79 | long result = values.nextLongValue();
80 | while (values.hasNextValue()) {
81 | result += values.nextLongValue();
82 | }
83 | return result;
84 | }
85 |
86 | public double runDouble(final Doubles values) {
87 | double result = values.nextDoubleValue();
88 | while (values.hasNextValue()) {
89 | result += values.nextDoubleValue();
90 | }
91 | return result;
92 | }
93 |
94 | public String toString() {
95 | return "sum";
96 | }
97 |
98 | }
99 |
100 | private static final class Min implements Aggregator {
101 |
102 | public long runLong(final Longs values) {
103 | long min = values.nextLongValue();
104 | while (values.hasNextValue()) {
105 | final long val = values.nextLongValue();
106 | if (val < min) {
107 | min = val;
108 | }
109 | }
110 | return min;
111 | }
112 |
113 | public double runDouble(final Doubles values) {
114 | double min = values.nextDoubleValue();
115 | while (values.hasNextValue()) {
116 | final double val = values.nextDoubleValue();
117 | if (val < min) {
118 | min = val;
119 | }
120 | }
121 | return min;
122 | }
123 |
124 | public String toString() {
125 | return "min";
126 | }
127 |
128 | }
129 |
130 | private static final class Max implements Aggregator {
131 |
132 | public long runLong(final Longs values) {
133 | long max = values.nextLongValue();
134 | while (values.hasNextValue()) {
135 | final long val = values.nextLongValue();
136 | if (val > max) {
137 | max = val;
138 | }
139 | }
140 | return max;
141 | }
142 |
143 | public double runDouble(final Doubles values) {
144 | double max = values.nextDoubleValue();
145 | while (values.hasNextValue()) {
146 | final double val = values.nextDoubleValue();
147 | if (val > max) {
148 | max = val;
149 | }
150 | }
151 | return max;
152 | }
153 |
154 | public String toString() {
155 | return "max";
156 | }
157 |
158 | }
159 |
160 | private static final class Avg implements Aggregator {
161 |
162 | public long runLong(final Longs values) {
163 | long result = values.nextLongValue();
164 | int n = 1;
165 | while (values.hasNextValue()) {
166 | result += values.nextLongValue();
167 | n++;
168 | }
169 | return result / n;
170 | }
171 |
172 | public double runDouble(final Doubles values) {
173 | double result = values.nextDoubleValue();
174 | int n = 1;
175 | while (values.hasNextValue()) {
176 | result += values.nextDoubleValue();
177 | n++;
178 | }
179 | return result / n;
180 | }
181 |
182 | public String toString() {
183 | return "avg";
184 | }
185 | }
186 |
187 | /**
188 | * Standard Deviation aggregator.
189 | * Can compute without storing all of the data points in memory at the same
190 | * time. This implementation is based upon a
191 | * paper by John
192 | * D. Cook , which itself is based upon a method that goes back to a 1962
193 | * paper by B. P. Welford and is presented in Donald Knuth's Art of
194 | * Computer Programming, Vol 2, page 232, 3rd edition
195 | */
196 | private static final class StdDev implements Aggregator {
197 |
198 | public long runLong(final Longs values) {
199 | double old_mean = values.nextLongValue();
200 |
201 | if (!values.hasNextValue()) {
202 | return 0;
203 | }
204 |
205 | long n = 2;
206 | double new_mean = 0;
207 | double variance = 0;
208 | do {
209 | final double x = values.nextLongValue();
210 | new_mean = old_mean + (x - old_mean) / n;
211 | variance += (x - old_mean) * (x - new_mean);
212 | old_mean = new_mean;
213 | n++;
214 | } while (values.hasNextValue());
215 |
216 | return (long) Math.sqrt(variance / (n - 1));
217 | }
218 |
219 | public double runDouble(final Doubles values) {
220 | double old_mean = values.nextDoubleValue();
221 |
222 | if (!values.hasNextValue()) {
223 | return 0;
224 | }
225 |
226 | long n = 2;
227 | double new_mean = 0;
228 | double variance = 0;
229 | do {
230 | final double x = values.nextDoubleValue();
231 | new_mean = old_mean + (x - old_mean) / n;
232 | variance += (x - old_mean) * (x - new_mean);
233 | old_mean = new_mean;
234 | n++;
235 | } while (values.hasNextValue());
236 |
237 | return Math.sqrt(variance / (n - 1));
238 | }
239 |
240 | public String toString() {
241 | return "dev";
242 | }
243 | }
244 |
245 | }
246 |
--------------------------------------------------------------------------------
/test/tsd/TestGraphHandler.java:
--------------------------------------------------------------------------------
1 | // This file is part of OpenTSDB.
2 | // Copyright (C) 2011-2012 The OpenTSDB Authors.
3 | //
4 | // This program is free software: you can redistribute it and/or modify it
5 | // under the terms of the GNU Lesser General Public License as published by
6 | // the Free Software Foundation, either version 2.1 of the License, or (at your
7 | // option) any later version. This program is distributed in the hope that it
8 | // will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty
9 | // of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser
10 | // General Public License for more details. You should have received a copy
11 | // of the GNU Lesser General Public License along with this program. If not,
12 | // see invokeMethod(GraphHandler.class, "staleCacheFile",
176 | query, end_time / 1000, max_age,
177 | cachedfile);
178 | }
179 |
180 | private static HttpQuery fakeHttpQuery() {
181 | final HttpQuery query = mock(HttpQuery.class);
182 | final Channel chan = fakeChannel();
183 | when(query.channel()).thenReturn(chan);
184 | return query;
185 | }
186 |
187 | private static Channel fakeChannel() {
188 | final Channel chan = mock(Channel.class);
189 | when(chan.toString()).thenReturn("[fake channel]");
190 | return chan;
191 | }
192 |
193 | private static File fakeFile(final String path) {
194 | final File file = mock(File.class);
195 | when(file.getPath()).thenReturn(path);
196 | when(file.toString()).thenReturn(path);
197 | return file;
198 | }
199 |
200 | }
201 |
--------------------------------------------------------------------------------
/src/stats/StatsCollector.java:
--------------------------------------------------------------------------------
1 | // This file is part of OpenTSDB.
2 | // Copyright (C) 2010-2012 The OpenTSDB Authors.
3 | //
4 | // This program is free software: you can redistribute it and/or modify it
5 | // under the terms of the GNU Lesser General Public License as published by
6 | // the Free Software Foundation, either version 2.1 of the License, or (at your
7 | // option) any later version. This program is distributed in the hope that it
8 | // will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty
9 | // of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser
10 | // General Public License for more details. You should have received a copy
11 | // of the GNU Lesser General Public License along with this program. If not,
12 | // see
26 | * Instances of this class are passed around to other classes to collect
27 | * their stats/metrics and do something with them (presumably send them
28 | * to a client).
29 | *
30 | * This class does not do any synchronization and is not thread-safe.
31 | */
32 | public abstract class StatsCollector {
33 |
34 | private static final Logger LOG =
35 | LoggerFactory.getLogger(StatsCollector.class);
36 |
37 | /** Prefix to add to every metric name, for example `tsd'. */
38 | private final String prefix;
39 |
40 | /** Extra tags to add to every data point emitted. */
41 | private HashMap extratags;
42 |
43 | /** Buffer used to build lines emitted. */
44 | private final StringBuilder buf = new StringBuilder();
45 |
46 | /**
47 | * Constructor.
48 | * @param prefix A prefix to add to every metric name, for example
49 | * `tsd'.
50 | */
51 | public StatsCollector(final String prefix) {
52 | this.prefix = prefix;
53 | }
54 |
55 | /**
56 | * Method to override to actually emit a data point.
57 | * @param datapoint A data point in a format suitable for a text
58 | * import.
59 | */
60 | public abstract void emit(String datapoint);
61 |
62 | /**
63 | * Records a data point.
64 | * @param name The name of the metric.
65 | * @param value The current value for that metric.
66 | */
67 | public final void record(final String name, final long value) {
68 | record(name, value, null);
69 | }
70 |
71 | /**
72 | * Records a data point.
73 | * @param name The name of the metric.
74 | * @param value The current value for that metric.
75 | */
76 | public final void record(final String name, final Number value) {
77 | record(name, value.longValue(), null);
78 | }
79 |
80 | /**
81 | * Records a data point.
82 | * @param name The name of the metric.
83 | * @param value The current value for that metric.
84 | * @param xtratag An extra tag ({@code name=value}) to add to those
85 | * data points (ignored if {@code null}).
86 | * @throws IllegalArgumentException if {@code xtratag != null} and it
87 | * doesn't follow the {@code name=value} format.
88 | */
89 | public final void record(final String name,
90 | final Number value,
91 | final String xtratag) {
92 | record(name, value.longValue(), xtratag);
93 | }
94 |
95 | /**
96 | * Records a number of data points from a {@link Histogram}.
97 | * @param name The name of the metric.
98 | * @param histo The histogram to collect data points from.
99 | * @param xtratag An extra tag ({@code name=value}) to add to those
100 | * data points (ignored if {@code null}).
101 | * @throws IllegalArgumentException if {@code xtratag != null} and it
102 | * doesn't follow the {@code name=value} format.
103 | */
104 | public final void record(final String name,
105 | final Histogram histo,
106 | final String xtratag) {
107 | record(name + "_50pct", histo.percentile(50), xtratag);
108 | record(name + "_75pct", histo.percentile(75), xtratag);
109 | record(name + "_90pct", histo.percentile(90), xtratag);
110 | record(name + "_95pct", histo.percentile(95), xtratag);
111 | }
112 |
113 | /**
114 | * Records a data point.
115 | * @param name The name of the metric.
116 | * @param value The current value for that metric.
117 | * @param xtratag An extra tag ({@code name=value}) to add to this
118 | * data point (ignored if {@code null}).
119 | * @throws IllegalArgumentException if {@code xtratag != null} and it
120 | * doesn't follow the {@code name=value} format.
121 | */
122 | public final void record(final String name,
123 | final long value,
124 | final String xtratag) {
125 | buf.setLength(0);
126 | buf.append(prefix).append(".")
127 | .append(name)
128 | .append(' ')
129 | .append(System.currentTimeMillis() / 1000)
130 | .append(' ')
131 | .append(value);
132 |
133 | if (xtratag != null) {
134 | if (xtratag.indexOf('=') != xtratag.lastIndexOf('=')) {
135 | throw new IllegalArgumentException("invalid xtratag: " + xtratag
136 | + " (multiple '=' signs), name=" + name + ", value=" + value);
137 | } else if (xtratag.indexOf('=') < 0) {
138 | throw new IllegalArgumentException("invalid xtratag: " + xtratag
139 | + " (missing '=' signs), name=" + name + ", value=" + value);
140 | }
141 | buf.append(' ').append(xtratag);
142 | }
143 |
144 | if (extratags != null) {
145 | for (final Map.Entry entry : extratags.entrySet()) {
146 | buf.append(' ').append(entry.getKey())
147 | .append('=').append(entry.getValue());
148 | }
149 | }
150 | buf.append('\n');
151 | emit(buf.toString());
152 | }
153 |
154 | /**
155 | * Adds a tag to all the subsequent data points recorded.
156 | *
157 | * All subsequent calls to one of the {@code record} methods will
158 | * associate the tag given to this method with the data point.
159 | *
160 | * This method can be called multiple times to associate multiple tags
161 | * with all the subsequent data points.
162 | * @param name The name of the tag.
163 | * @param value The value of the tag.
164 | * @throws IllegalArgumentException if the name or the value are empty
165 | * or otherwise invalid.
166 | * @see #clearExtraTag
167 | */
168 | public final void addExtraTag(final String name, final String value) {
169 | if (name.length() <= 0) {
170 | throw new IllegalArgumentException("empty tag name, value=" + value);
171 | } else if (value.length() <= 0) {
172 | throw new IllegalArgumentException("empty value, tag name=" + name);
173 | } else if (name.indexOf('=') != -1) {
174 | throw new IllegalArgumentException("tag name contains `=': " + name
175 | + " (value = " + value + ')');
176 | } else if (value.indexOf('=') != -1) {
177 | throw new IllegalArgumentException("tag value contains `=': " + value
178 | + " (name = " + name + ')');
179 | }
180 | if (extratags == null) {
181 | extratags = new HashMap();
182 | }
183 | extratags.put(name, value);
184 | }
185 |
186 | /**
187 | * Adds a {@code host=hostname} tag.
188 | *
189 | * This uses {@link InetAddress#getLocalHost} to find the hostname of the
190 | * current host. If the hostname cannot be looked up, {@code (unknown)}
191 | * is used instead.
192 | */
193 | public final void addHostTag() {
194 | try {
195 | addExtraTag("host", InetAddress.getLocalHost().getHostName());
196 | } catch (UnknownHostException x) {
197 | LOG.error("WTF? Can't find hostname for localhost!", x);
198 | addExtraTag("host", "(unknown)");
199 | }
200 | }
201 |
202 | /**
203 | * Clears a tag added using {@link #addExtraTag addExtraTag}.
204 | * @param name The name of the tag to remove from the set of extra
205 | * tags.
206 | * @throws IllegalStateException if there's no extra tag currently
207 | * recorded.
208 | * @throws IllegalArgumentException if the given name isn't in the
209 | * set of extra tags currently recorded.
210 | * @see #addExtraTag
211 | */
212 | public final void clearExtraTag(final String name) {
213 | if (extratags == null) {
214 | throw new IllegalStateException("no extra tags added");
215 | }
216 | if (extratags.get(name) == null) {
217 | throw new IllegalArgumentException("tag '" + name
218 | + "' not in" + extratags);
219 | }
220 | extratags.remove(name);
221 | }
222 |
223 | }
224 |
--------------------------------------------------------------------------------
/src/tools/TSDMain.java:
--------------------------------------------------------------------------------
1 | // This file is part of OpenTSDB.
2 | // Copyright (C) 2010-2012 The OpenTSDB Authors.
3 | //
4 | // This program is free software: you can redistribute it and/or modify it
5 | // under the terms of the GNU Lesser General Public License as published by
6 | // the Free Software Foundation, either version 2.1 of the License, or (at your
7 | // option) any later version. This program is distributed in the hope that it
8 | // will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty
9 | // of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser
10 | // General Public License for more details. You should have received a copy
11 | // of the GNU Lesser General Public License along with this program. If not,
12 | // see {
107 | public Object call(final Exception arg) {
108 | if (arg instanceof PleaseThrottleException) {
109 | final PleaseThrottleException e = (PleaseThrottleException) arg;
110 | LOG.warn("Need to throttle, HBase isn't keeping up.", e);
111 | throttle = true;
112 | final HBaseRpc rpc = e.getFailedRpc();
113 | if (rpc instanceof PutRequest) {
114 | client.put((PutRequest) rpc); // Don't lose edits.
115 | }
116 | return null;
117 | }
118 | LOG.error("Exception caught while processing file "
119 | + path, arg);
120 | System.exit(2);
121 | return arg;
122 | }
123 | public String toString() {
124 | return "importFile errback";
125 | }
126 | };
127 | final Errback errback = new Errback();
128 | while ((line = in.readLine()) != null) {
129 | final String[] words = Tags.splitString(line, ' ');
130 | final String metric = words[0];
131 | if (metric.length() <= 0) {
132 | throw new RuntimeException("invalid metric: " + metric);
133 | }
134 | final long timestamp = Tags.parseLong(words[1]);
135 | if (timestamp <= 0) {
136 | throw new RuntimeException("invalid timestamp: " + timestamp);
137 | }
138 | final String value = words[2];
139 | if (value.length() <= 0) {
140 | throw new RuntimeException("invalid value: " + value);
141 | }
142 | final HashMap tags = new HashMap();
143 | for (int i = 3; i < words.length; i++) {
144 | if (!words[i].isEmpty()) {
145 | Tags.parse(tags, words[i]);
146 | }
147 | }
148 | final WritableDataPoints dp = getDataPoints(tsdb, metric, tags);
149 | Deferred d;
150 | if (Tags.looksLikeInteger(value)) {
151 | d = dp.addPoint(timestamp, Tags.parseLong(value));
152 | } else { // floating point value
153 | d = dp.addPoint(timestamp, Float.parseFloat(value));
154 | }
155 | d.addErrback(errback);
156 | points++;
157 | if (points % 1000000 == 0) {
158 | final long now = System.nanoTime();
159 | ping_start_time = (now - ping_start_time) / 1000000;
160 | LOG.info(String.format("... %d data points in %dms (%.1f points/s)",
161 | points, ping_start_time,
162 | (1000000 * 1000.0 / ping_start_time)));
163 | ping_start_time = now;
164 | }
165 | if (throttle) {
166 | LOG.info("Throttling...");
167 | long throttle_time = System.nanoTime();
168 | try {
169 | d.joinUninterruptibly();
170 | } catch (Exception e) {
171 | throw new RuntimeException("Should never happen", e);
172 | }
173 | throttle_time = System.nanoTime() - throttle_time;
174 | if (throttle_time < 1000000000L) {
175 | LOG.info("Got throttled for only " + throttle_time + "ns, sleeping a bit now");
176 | try { Thread.sleep(1000); } catch (InterruptedException e) { throw new RuntimeException("interrupted", e); }
177 | }
178 | LOG.info("Done throttling...");
179 | throttle = false;
180 | }
181 | }
182 | } catch (RuntimeException e) {
183 | LOG.error("Exception caught while processing file "
184 | + path + " line=" + line);
185 | throw e;
186 | } finally {
187 | in.close();
188 | }
189 | final long time_delta = (System.nanoTime() - start_time) / 1000000;
190 | LOG.info(String.format("Processed %s in %d ms, %d data points"
191 | + " (%.1f points/s)",
192 | path, time_delta, points,
193 | (points * 1000.0 / time_delta)));
194 | return points;
195 | }
196 |
197 | /**
198 | * Opens a file for reading, handling gzipped files.
199 | * @param path The file to open.
200 | * @return A buffered reader to read the file, decompressing it if needed.
201 | * @throws IOException when shit happens.
202 | */
203 | private static BufferedReader open(final String path) throws IOException {
204 | InputStream is = new FileInputStream(path);
205 | if (path.endsWith(".gz")) {
206 | is = new GZIPInputStream(is);
207 | }
208 | // I <3 Java's IO library.
209 | return new BufferedReader(new InputStreamReader(is));
210 | }
211 |
212 | private static final HashMap datapoints =
213 | new HashMap();
214 |
215 | private static
216 | WritableDataPoints getDataPoints(final TSDB tsdb,
217 | final String metric,
218 | final HashMap tags) {
219 | final String key = metric + tags;
220 | WritableDataPoints dp = datapoints.get(key);
221 | if (dp != null) {
222 | return dp;
223 | }
224 | dp = tsdb.newDataPoints();
225 | dp.setSeries(metric, tags);
226 | dp.setBatchImport(true);
227 | datapoints.put(key, dp);
228 | return dp;
229 | }
230 |
231 | }
232 |
--------------------------------------------------------------------------------
/src/tools/DumpSeries.java:
--------------------------------------------------------------------------------
1 | // This file is part of OpenTSDB.
2 | // Copyright (C) 2010-2012 The OpenTSDB Authors.
3 | //
4 | // This program is free software: you can redistribute it and/or modify it
5 | // under the terms of the GNU Lesser General Public License as published by
6 | // the Free Software Foundation, either version 2.1 of the License, or (at your
7 | // option) any later version. This program is distributed in the hope that it
8 | // will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty
9 | // of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser
10 | // General Public License for more details. You should have received a copy
11 | // of the GNU Lesser General Public License along with this program. If not,
12 | // see queries = new ArrayList();
89 | CliQuery.parseCommandLineQuery(args, tsdb, queries, null, null);
90 |
91 | final StringBuilder buf = new StringBuilder();
92 | for (final Query query : queries) {
93 | final Scanner scanner = Internal.getScanner(query);
94 | ArrayList> rows;
95 | while ((rows = scanner.nextRows().joinUninterruptibly()) != null) {
96 | for (final ArrayList row : rows) {
97 | buf.setLength(0);
98 | final byte[] key = row.get(0).key();
99 | final long base_time = Internal.baseTime(tsdb, key);
100 | final String metric = Internal.metricName(tsdb, key);
101 | // Print the row key.
102 | if (!importformat) {
103 | buf.append(Arrays.toString(key))
104 | .append(' ')
105 | .append(metric)
106 | .append(' ')
107 | .append(base_time)
108 | .append(" (").append(date(base_time)).append(") ");
109 | try {
110 | buf.append(Internal.getTags(tsdb, key));
111 | } catch (RuntimeException e) {
112 | buf.append(e.getClass().getName() + ": " + e.getMessage());
113 | }
114 | buf.append('\n');
115 | System.out.print(buf);
116 | }
117 |
118 | // Print individual cells.
119 | buf.setLength(0);
120 | if (!importformat) {
121 | buf.append(" ");
122 | }
123 | for (final KeyValue kv : row) {
124 | // Discard everything or keep initial spaces.
125 | buf.setLength(importformat ? 0 : 2);
126 | formatKeyValue(buf, tsdb, importformat, kv, base_time, metric);
127 | buf.append('\n');
128 | System.out.print(buf);
129 | }
130 |
131 | if (delete) {
132 | final DeleteRequest del = new DeleteRequest(table, key);
133 | client.delete(del);
134 | }
135 | }
136 | }
137 | }
138 | }
139 |
140 | static void formatKeyValue(final StringBuilder buf,
141 | final TSDB tsdb,
142 | final KeyValue kv,
143 | final long base_time) {
144 | formatKeyValue(buf, tsdb, true, kv, base_time,
145 | Internal.metricName(tsdb, kv.key()));
146 | }
147 |
148 | private static void formatKeyValue(final StringBuilder buf,
149 | final TSDB tsdb,
150 | final boolean importformat,
151 | final KeyValue kv,
152 | final long base_time,
153 | final String metric) {
154 | if (importformat) {
155 | buf.append(metric).append(' ');
156 | }
157 | final byte[] qualifier = kv.qualifier();
158 | final byte[] cell = kv.value();
159 | if (qualifier.length != 2 && cell[cell.length - 1] != 0) {
160 | throw new IllegalDataException("Don't know how to read this value:"
161 | + Arrays.toString(cell) + " found in " + kv
162 | + " -- this compacted value might have been written by a future"
163 | + " version of OpenTSDB, or could be corrupt.");
164 | }
165 | final int nvalues = qualifier.length / 2;
166 | final boolean multi_val = nvalues != 1 && !importformat;
167 | if (multi_val) {
168 | buf.append(Arrays.toString(qualifier))
169 | .append(' ').append(Arrays.toString(cell))
170 | .append(" = ").append(nvalues).append(" values:");
171 | }
172 |
173 | final String tags;
174 | if (importformat) {
175 | final StringBuilder tagsbuf = new StringBuilder();
176 | for (final Map.Entry tag
177 | : Internal.getTags(tsdb, kv.key()).entrySet()) {
178 | tagsbuf.append(' ').append(tag.getKey())
179 | .append('=').append(tag.getValue());
180 | }
181 | tags = tagsbuf.toString();
182 | } else {
183 | tags = null;
184 | }
185 |
186 | int value_offset = 0;
187 | for (int i = 0; i < nvalues; i++) {
188 | if (multi_val) {
189 | buf.append("\n ");
190 | }
191 | final short qual = Bytes.getShort(qualifier, i * 2);
192 | final byte flags = (byte) qual;
193 | final int value_len = (flags & 0x7) + 1;
194 | final short delta = (short) ((0x0000FFFF & qual) >>> 4);
195 | if (importformat) {
196 | buf.append(base_time + delta).append(' ');
197 | } else {
198 | final byte[] v = multi_val
199 | ? Arrays.copyOfRange(cell, value_offset, value_offset + value_len)
200 | : cell;
201 | buf.append(Arrays.toString(Bytes.fromShort(qual)))
202 | .append(' ')
203 | .append(Arrays.toString(v))
204 | .append('\t')
205 | .append(delta)
206 | .append('\t');
207 | }
208 | if ((qual & 0x8) == 0x8) {
209 | if (cell.length == 8 && value_len == 4
210 | && cell[0] == 0 && cell[1] == 0 && cell[2] == 0 && cell[3] == 0) {
211 | // Incorrect encoded floating point value.
212 | // See CompactionQueue.fixFloatingPointValue() for more details.
213 | value_offset += 4;
214 | }
215 | buf.append(importformat ? "" : "f ")
216 | .append(Internal.extractFloatingPointValue(cell, value_offset, flags));
217 | } else {
218 | buf.append(importformat ? "" : "l ")
219 | .append(Internal.extractIntegerValue(cell, value_offset, flags));
220 | }
221 | if (importformat) {
222 | buf.append(tags);
223 | if (nvalues > 1 && i + 1 < nvalues) {
224 | buf.append('\n').append(metric).append(' ');
225 | }
226 | } else {
227 | buf.append('\t')
228 | .append(base_time + delta)
229 | .append(" (").append(date(base_time + delta)).append(')');
230 | }
231 | value_offset += value_len;
232 | }
233 | }
234 |
235 | /** Transforms a UNIX timestamp into a human readable date. */
236 | static String date(final long timestamp) {
237 | return new Date(timestamp * 1000).toString();
238 | }
239 |
240 | }
241 |
--------------------------------------------------------------------------------
/src/tools/CliQuery.java:
--------------------------------------------------------------------------------
1 | // This file is part of OpenTSDB.
2 | // Copyright (C) 2010-2012 The OpenTSDB Authors.
3 | //
4 | // This program is free software: you can redistribute it and/or modify it
5 | // under the terms of the GNU Lesser General Public License as published by
6 | // the Free Software Foundation, either version 2.1 of the License, or (at your
7 | // option) any later version. This program is distributed in the hope that it
8 | // will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty
9 | // of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser
10 | // General Public License for more details. You should have received a copy
11 | // of the GNU Lesser General Public License along with this program. If not,
12 | // see [queries...]\n"
45 | + "A query has the form:\n"
46 | + " FUNC [rate] [downsample FUNC N] SERIES [TAGS]\n"
47 | + "For example:\n"
48 | + " 2010/03/11-20:57 sum my.awsum.metric host=blah"
49 | + " sum some.other.metric host=blah state=foo\n"
50 | + "Dates must follow this format: [YYYY/MM/DD-]HH:MM[:SS]\n"
51 | + "Supported values for FUNC: " + Aggregators.set()
52 | + "\nGnuplot options are of the form: +option=value");
53 | if (argp != null) {
54 | System.err.print(argp.usage());
55 | }
56 | System.exit(retval);
57 | }
58 |
59 | /** Parses the date in argument and returns a UNIX timestamp in seconds. */
60 | private static long parseDate(final String s) {
61 | SimpleDateFormat format;
62 | switch (s.length()) {
63 | case 5:
64 | format = new SimpleDateFormat("HH:mm");
65 | break;
66 | case 8:
67 | format = new SimpleDateFormat("HH:mm:ss");
68 | break;
69 | case 10:
70 | format = new SimpleDateFormat("yyyy/MM/dd");
71 | break;
72 | case 16:
73 | format = new SimpleDateFormat("yyyy/MM/dd-HH:mm");
74 | break;
75 | case 19:
76 | format = new SimpleDateFormat("yyyy/MM/dd-HH:mm:ss");
77 | break;
78 | default:
79 | usage(null, "Invalid date: " + s, 3);
80 | return -1; // Never executed as usage() exits.
81 | }
82 | try {
83 | return format.parse(s).getTime() / 1000;
84 | } catch (ParseException e) {
85 | usage(null, "Invalid date: " + s, 3);
86 | return -1; // Never executed as usage() exits.
87 | }
88 | }
89 |
90 | public static void main(String[] args) throws IOException {
91 | ArgP argp = new ArgP();
92 | CliOptions.addCommon(argp);
93 | CliOptions.addVerbose(argp);
94 | argp.addOption("--graph", "BASEPATH",
95 | "Output data points to a set of files for gnuplot."
96 | + " The path of the output files will start with"
97 | + " BASEPATH.");
98 | args = CliOptions.parse(argp, args);
99 | if (args == null) {
100 | usage(argp, "Invalid usage.", 1);
101 | } else if (args.length < 3) {
102 | usage(argp, "Not enough arguments.", 2);
103 | }
104 |
105 | final HBaseClient client = CliOptions.clientFromOptions(argp);
106 | final TSDB tsdb = new TSDB(client, argp.get("--table", "tsdb"),
107 | argp.get("--uidtable", "tsdb-uid"));
108 | final String basepath = argp.get("--graph");
109 | argp = null;
110 |
111 | Plot plot = null;
112 | try {
113 | plot = doQuery(tsdb, args, basepath != null);
114 | } finally {
115 | try {
116 | tsdb.shutdown().joinUninterruptibly();
117 | } catch (Exception e) {
118 | LOG.error("Unexpected exception", e);
119 | System.exit(1);
120 | }
121 | }
122 |
123 | if (plot != null) {
124 | try {
125 | final int npoints = plot.dumpToFiles(basepath);
126 | LOG.info("Wrote " + npoints + " for Gnuplot");
127 | } catch (IOException e) {
128 | LOG.error("Failed to write the Gnuplot file under " + basepath, e);
129 | System.exit(1);
130 | }
131 | }
132 | }
133 |
134 | private static Plot doQuery(final TSDB tsdb,
135 | final String args[],
136 | final boolean want_plot) {
137 | final ArrayList plotparams = new ArrayList();
138 | final ArrayList queries = new ArrayList();
139 | final ArrayList plotoptions = new ArrayList();
140 | parseCommandLineQuery(args, tsdb, queries, plotparams, plotoptions);
141 | if (queries.isEmpty()) {
142 | usage(null, "Not enough arguments, need at least one query.", 2);
143 | }
144 |
145 | final Plot plot = (want_plot ? new Plot(queries.get(0).getStartTime(),
146 | queries.get(0).getEndTime())
147 | : null);
148 | if (want_plot) {
149 | plot.setParams(parsePlotParams(plotparams));
150 | }
151 | final int nqueries = queries.size();
152 | for (int i = 0; i < nqueries; i++) {
153 | // TODO(tsuna): Optimization: run each query in parallel.
154 | final StringBuilder buf = want_plot ? null : new StringBuilder();
155 | for (final DataPoints datapoints : queries.get(i).run()) {
156 | if (want_plot) {
157 | plot.add(datapoints, plotoptions.get(i));
158 | } else {
159 | final String metric = datapoints.metricName();
160 | final String tagz = datapoints.getTags().toString();
161 | for (final DataPoint datapoint : datapoints) {
162 | buf.append(metric)
163 | .append(' ')
164 | .append(datapoint.timestamp())
165 | .append(' ');
166 | if (datapoint.isInteger()) {
167 | buf.append(datapoint.longValue());
168 | } else {
169 | buf.append(String.format("%f", datapoint.doubleValue()));
170 | }
171 | buf.append(' ').append(tagz).append('\n');
172 | System.out.print(buf);
173 | buf.delete(0, buf.length());
174 | }
175 | }
176 | }
177 | }
178 | return plot;
179 | }
180 |
181 | /**
182 | * Parses the query from the command lines.
183 | * @param args The command line arguments.
184 | * @param tsdb The TSDB to use.
185 | * @param queries The list in which {@link Query}s will be appended.
186 | * @param plotparams The list in which global plot parameters will be
187 | * appended. Ignored if {@code null}.
188 | * @param plotoptions The list in which per-line plot options will be
189 | * appended. Ignored if {@code null}.
190 | */
191 | static void parseCommandLineQuery(final String[] args,
192 | final TSDB tsdb,
193 | final ArrayList queries,
194 | final ArrayList plotparams,
195 | final ArrayList plotoptions) {
196 | final long start_ts = parseDate(args[0]);
197 | final long end_ts = (args.length > 3
198 | && (args[1].charAt(0) != '+'
199 | && (args[1].indexOf(':') >= 0
200 | || args[1].indexOf('/') >= 0))
201 | ? parseDate(args[1]) : -1);
202 |
203 | int i = end_ts < 0 ? 1 : 2;
204 | while (i < args.length && args[i].charAt(0) == '+') {
205 | if (plotparams != null) {
206 | plotparams.add(args[i]);
207 | }
208 | i++;
209 | }
210 |
211 | while (i < args.length) {
212 | final Aggregator agg = Aggregators.get(args[i++]);
213 | final boolean rate = args[i].equals("rate");
214 | if (rate) {
215 | i++;
216 | }
217 | final boolean downsample = args[i].equals("downsample");
218 | if (downsample) {
219 | i++;
220 | }
221 | final int interval = downsample ? Integer.parseInt(args[i++]) : 0;
222 | final Aggregator sampler = downsample ? Aggregators.get(args[i++]) : null;
223 | final String metric = args[i++];
224 | final HashMap tags = new HashMap();
225 | while (i < args.length && args[i].indexOf(' ', 1) < 0
226 | && args[i].indexOf('=', 1) > 0) {
227 | Tags.parse(tags, args[i++]);
228 | }
229 | if (i < args.length && args[i].indexOf(' ', 1) > 0) {
230 | plotoptions.add(args[i++]);
231 | }
232 | final Query query = tsdb.newQuery();
233 | query.setStartTime(start_ts);
234 | if (end_ts > 0) {
235 | query.setEndTime(end_ts);
236 | }
237 | query.setTimeSeries(metric, tags, agg, rate);
238 | if (downsample) {
239 | query.downsample(interval, sampler);
240 | }
241 | queries.add(query);
242 | }
243 | }
244 |
245 | private static HashMap parsePlotParams(final ArrayList params) {
246 | final HashMap result =
247 | new HashMap(params.size());
248 | for (final String param : params) {
249 | Tags.parse(result, param.substring(1));
250 | }
251 | return result;
252 | }
253 |
254 | }
255 |
--------------------------------------------------------------------------------