├── project ├── build.properties ├── plugins.sbt └── KukulcanPackPlugin.scala ├── python ├── pykukulcan │ ├── repl │ │ ├── requirements.txt │ │ ├── __main__.py │ │ └── __init__.py │ ├── version.py │ └── __init__.py ├── requirements.txt └── setup.py ├── docs └── img │ ├── kksql.png │ ├── kconnect.png │ ├── kstreams.png │ ├── kukulcan.png │ └── kschema-registry.png ├── kukulcan-repl └── src │ ├── main │ ├── resources │ │ └── log4j.properties │ ├── java │ │ └── com │ │ │ └── github │ │ │ └── mmolimar │ │ │ └── kukulcan │ │ │ ├── repl │ │ │ └── JKukulcanRepl.java │ │ │ └── java │ │ │ └── Kukulcan.java │ └── scala │ │ └── com │ │ └── github │ │ └── mmolimar │ │ └── kukulcan │ │ ├── repl │ │ ├── KukulcanRepl.scala │ │ ├── KApi.scala │ │ └── package.scala │ │ └── package.scala │ └── test │ └── scala │ └── com │ └── github │ └── mmolimar │ └── kukulcan │ └── repl │ └── KukulcanReplSpec.scala ├── bin ├── pykukulcan ├── kukulcan ├── jkukulcan ├── pykukulcan.cmd ├── kukulcan-amm ├── find-kukulcan-home.cmd ├── kukulcan.cmd ├── jkukulcan.cmd └── find-kukulcan-home ├── kukulcan-api └── src │ ├── test │ ├── resources │ │ └── log4j.properties │ └── scala │ │ └── com │ │ └── github │ │ └── mmolimar │ │ └── kukulcan │ │ ├── KukulcanApiTestHarness.scala │ │ ├── KAdminMetricsSpec.scala │ │ ├── KProducerSpec.scala │ │ ├── KConsumerSpec.scala │ │ ├── KStreamsSpec.scala │ │ ├── KAdminAclsSpec.scala │ │ ├── KKsqlSpec.scala │ │ ├── KAdminTopicsSpec.scala │ │ ├── KConnectSpec.scala │ │ ├── KAdminConfigsSpec.scala │ │ └── KSchemaRegistrySpec.scala │ └── main │ ├── scala │ └── com │ │ └── github │ │ └── mmolimar │ │ └── kukulcan │ │ ├── KProducer.scala │ │ ├── KConsumer.scala │ │ ├── KKsql.scala │ │ └── KSchemaRegistry.scala │ └── java │ └── com │ └── github │ └── mmolimar │ └── kukulcan │ └── java │ ├── KUtils.java │ ├── KProducer.java │ ├── KConsumer.java │ ├── KStreams.java │ ├── KConnect.java │ ├── KKsql.java │ └── KSchemaRegistry.java ├── .gitignore ├── .circleci └── config.yml ├── config ├── connect.properties ├── schema-registry.properties ├── ksql.properties ├── streams.properties └── admin.properties ├── README.md └── LICENSE /project/build.properties: -------------------------------------------------------------------------------- 1 | sbt.version = 1.4.7 2 | -------------------------------------------------------------------------------- /python/pykukulcan/repl/requirements.txt: -------------------------------------------------------------------------------- 1 | py4j>=0.10.9.2 2 | -------------------------------------------------------------------------------- /python/pykukulcan/version.py: -------------------------------------------------------------------------------- 1 | __version__ = '0.2.0' 2 | -------------------------------------------------------------------------------- /python/requirements.txt: -------------------------------------------------------------------------------- 1 | -r pykukulcan/repl/requirements.txt 2 | -------------------------------------------------------------------------------- /docs/img/kksql.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mmolimar/kukulcan/HEAD/docs/img/kksql.png -------------------------------------------------------------------------------- /docs/img/kconnect.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mmolimar/kukulcan/HEAD/docs/img/kconnect.png -------------------------------------------------------------------------------- /docs/img/kstreams.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mmolimar/kukulcan/HEAD/docs/img/kstreams.png -------------------------------------------------------------------------------- /docs/img/kukulcan.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mmolimar/kukulcan/HEAD/docs/img/kukulcan.png -------------------------------------------------------------------------------- /docs/img/kschema-registry.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mmolimar/kukulcan/HEAD/docs/img/kschema-registry.png -------------------------------------------------------------------------------- /project/plugins.sbt: -------------------------------------------------------------------------------- 1 | addSbtPlugin("org.xerial.sbt" % "sbt-pack" % "0.13") 2 | addSbtPlugin("com.github.sbt" % "sbt-jacoco" % "3.3.0") 3 | -------------------------------------------------------------------------------- /python/pykukulcan/__init__.py: -------------------------------------------------------------------------------- 1 | from pykukulcan.repl import PyKukulcanRepl 2 | from . import version 3 | 4 | __all__ = ['PyKukulcanRepl'] 5 | 6 | __version__ = version.__version__ 7 | -------------------------------------------------------------------------------- /kukulcan-repl/src/main/resources/log4j.properties: -------------------------------------------------------------------------------- 1 | log4j.rootLogger=ERROR, stdout 2 | 3 | log4j.appender.stdout=org.apache.log4j.ConsoleAppender 4 | log4j.appender.stdout.layout=org.apache.log4j.PatternLayout 5 | log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m %n 6 | -------------------------------------------------------------------------------- /bin/pykukulcan: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | source "$(dirname "$0")"/find-kukulcan-home 4 | 5 | if [[ -z "$KUKULCAN_PYTHON" ]]; then 6 | KUKULCAN_PYTHON=python 7 | fi 8 | 9 | eval exec "\"$KUKULCAN_PYTHON\"" -i -m pykukulcan.repl --classpath "\"${KUKULCAN_CLASSPATH}\"" 10 | 11 | exit $? 12 | -------------------------------------------------------------------------------- /kukulcan-api/src/test/resources/log4j.properties: -------------------------------------------------------------------------------- 1 | log4j.rootLogger=off 2 | 3 | log4j.appender.stdout=org.apache.log4j.ConsoleAppender 4 | log4j.appender.stdout.layout=org.apache.log4j.PatternLayout 5 | 6 | # Pattern to output the caller's file name and line number. 7 | log4j.appender.stdout.layout.ConversionPattern=%5p [%t] (%F:%L) - %m%n 8 | -------------------------------------------------------------------------------- /bin/kukulcan: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | source "$(dirname "$0")"/find-kukulcan-home 4 | 5 | for arg do 6 | shift 7 | case $arg in 8 | -D*) KUKULCAN_OPTS="$KUKULCAN_OPTS $arg" ;; 9 | *) set -- "$@" "$arg" ;; 10 | esac 11 | done 12 | 13 | eval exec "\"$JAVA_EXEC\"" ${KUKULCAN_OPTS} -cp "\"${KUKULCAN_CLASSPATH}\"" \ 14 | com.github.mmolimar.kukulcan.repl.KukulcanRepl 15 | 16 | exit $? 17 | -------------------------------------------------------------------------------- /bin/jkukulcan: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | source "$(dirname "$0")"/find-kukulcan-home 4 | 5 | KUKULCAN_OPTS="--add-exports=jdk.jshell/jdk.internal.jshell.tool=ALL-UNNAMED" 6 | 7 | for arg do 8 | shift 9 | case $arg in 10 | -D*) KUKULCAN_OPTS="$KUKULCAN_OPTS $arg" ;; 11 | *) set -- "$@" "$arg" ;; 12 | esac 13 | done 14 | 15 | eval exec "\"$JAVA_EXEC\"" ${KUKULCAN_OPTS} -cp "\"${KUKULCAN_CLASSPATH}\"" \ 16 | com.github.mmolimar.kukulcan.repl.JKukulcanRepl \ 17 | --class-path "\"$(find "$(echo "$KUKULCAN_CLASSPATH" | sed 's/\/\*//g')" | tr '\n' ':')\"" 18 | 19 | exit $? 20 | -------------------------------------------------------------------------------- /bin/pykukulcan.cmd: -------------------------------------------------------------------------------- 1 | @echo off 2 | 3 | set ERROR_CODE=0 4 | 5 | call "%~dp0find-kukulcan-home.cmd" 6 | 7 | if "%KUKULCAN_PYTHON%" == "" set KUKULCAN_PYTHON=python 8 | set /P KUKULCAN_CLASSPATH=<%TMP%\KUKULCAN_CLASSPATH 9 | 10 | %KUKULCAN_PYTHON% -i -m pykukulcan.repl --classpath "%KUKULCAN_CLASSPATH%" 11 | 12 | if ERRORLEVEL 1 goto error 13 | goto end 14 | 15 | :error 16 | if "%OS%"=="Windows_NT" @endlocal 17 | set ERROR_CODE=1 18 | 19 | :end 20 | if "%OS%"=="Windows_NT" goto endNT 21 | set JAVA_EXEC= 22 | set KUKULCAN_OPTS= 23 | set KUKULCAN_CLASSPATH= 24 | goto postExec 25 | 26 | :endNT 27 | @endlocal 28 | 29 | :postExec 30 | exit /B %ERROR_CODE% 31 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # use glob syntax. 2 | syntax: glob 3 | *.ser 4 | *.class 5 | *~ 6 | *.bak 7 | *.off 8 | *.old 9 | .DS_Store 10 | 11 | # IDE 12 | .settings 13 | .classpath 14 | .project 15 | .manager 16 | .idea 17 | *.iml 18 | .pydevproject 19 | .coverage 20 | 21 | # building 22 | .cache 23 | target 24 | build 25 | libs 26 | venv 27 | null 28 | out 29 | tmp 30 | temp 31 | test-output 32 | build.log 33 | sbt.json 34 | python/*.egg-info 35 | python/setup.cfg 36 | __pycache__ 37 | 38 | # other scm 39 | .svn 40 | .CVS 41 | .hg* 42 | 43 | # switch to regexp syntax. 44 | # syntax: regexp 45 | # ^\.pc/ 46 | 47 | # Documentation autogenerated 48 | javadoc 49 | apidocs 50 | -------------------------------------------------------------------------------- /kukulcan-api/src/test/scala/com/github/mmolimar/kukulcan/KukulcanApiTestHarness.scala: -------------------------------------------------------------------------------- 1 | package com.github.mmolimar.kukulcan 2 | 3 | import org.scalatest.matchers.should.Matchers 4 | import org.scalatest.wordspec.AnyWordSpecLike 5 | 6 | abstract class KukulcanApiTestHarness extends AnyWordSpecLike with Matchers { 7 | 8 | def apiClass: Class[_] 9 | 10 | def execJavaTests(): Unit 11 | 12 | def execScalaTests(): Unit 13 | 14 | s"a ${apiClass.getSimpleName}" when { 15 | "running tests for the Scala API" should { 16 | "validate its methods" in execScalaTests() 17 | } 18 | 19 | "running tests for the Java API" should { 20 | "validate its methods" in execJavaTests() 21 | } 22 | } 23 | 24 | } 25 | -------------------------------------------------------------------------------- /bin/kukulcan-amm: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env amm 2 | 3 | val kukulcanVersion = "0.2.0" 4 | val predefCode = 5 | s""" 6 | |import coursierapi.MavenRepository 7 | | 8 | |interp.repositories.update( 9 | | List( 10 | | coursierapi.Repository.ivy2Local, 11 | | coursierapi.Repository.central, 12 | | MavenRepository.of("https://packages.confluent.io/maven/"), 13 | | MavenRepository.of("https://jitpack.io/")) 14 | |) 15 | |@ 16 | | 17 | |import $$ivy.`com.github.mmolimar::kukulcan-repl:$kukulcanVersion` 18 | |import com.github.mmolimar.kukulcan 19 | |import os._ 20 | | 21 | |print(kukulcan.repl.banner) 22 | | 23 | |""".stripMargin 24 | 25 | ammonite.Main.main(Array( 26 | "--banner", "", 27 | "--predef-code", predefCode, 28 | )) 29 | -------------------------------------------------------------------------------- /.circleci/config.yml: -------------------------------------------------------------------------------- 1 | version: 2.1 2 | jobs: 3 | build: 4 | docker: 5 | - image: circleci/openjdk:11-jdk 6 | 7 | working_directory: ~/kukulcan 8 | 9 | environment: 10 | JVM_OPTS: -Xmx3200m 11 | TERM: dumb 12 | 13 | steps: 14 | - checkout 15 | - restore_cache: 16 | keys: 17 | - v1-dependencies-{{ checksum "build.sbt" }} 18 | - v1-dependencies- 19 | - run: 20 | name: Compile and test 21 | command: sbt clean compile test 22 | - run: 23 | name: Code coverage 24 | command: sbt jacocoAggregate jacocoAggregateReport 25 | - run: 26 | name: Build and publish 27 | command: sbt kukulcan publishLocal 28 | - save_cache: 29 | paths: 30 | - ~/.m2 31 | key: v1-dependencies--{{ checksum "build.sbt" }} 32 | -------------------------------------------------------------------------------- /bin/find-kukulcan-home.cmd: -------------------------------------------------------------------------------- 1 | @echo off 2 | 3 | if "%OS%"=="Windows_NT" @setlocal 4 | if not "%JAVA_HOME%" == "" goto OkJHome 5 | 6 | for /f %%j in ("java.exe") do ( 7 | set JAVA_EXEC="%%~$PATH:j" 8 | goto setVars 9 | ) 10 | 11 | :OkJHome 12 | if exist "%JAVA_HOME%\bin\java.exe" ( 13 | set JAVA_EXEC="%JAVA_HOME%\bin\java.exe" 14 | goto setVars 15 | ) 16 | 17 | echo. 1>&2 18 | echo ERROR: JAVA_HOME is set to an invalid directory. 1>&2 19 | echo JAVA_HOME = %JAVA_HOME% 1>&2 20 | echo Please set the JAVA_HOME variable in your environment to match the 1>&2 21 | echo location of your Java installation 1>&2 22 | echo. 1>&2 23 | goto error 24 | 25 | :setVars 26 | if "%KUKULCAN_HOME%" == "" (set KUKULCAN_HOME=%~dp0..) 27 | set KUKULCAN_LIBS_DIR=%KUKULCAN_HOME%\libs 28 | set KUKULCAN_CLASSPATH=%KUKULCAN_LIBS_DIR%\* 29 | 30 | echo %JAVA_EXEC%>%TMP%\JAVA_EXEC 31 | echo %KUKULCAN_CLASSPATH%>%TMP%\KUKULCAN_CLASSPATH 32 | -------------------------------------------------------------------------------- /bin/kukulcan.cmd: -------------------------------------------------------------------------------- 1 | @echo off 2 | 3 | set ERROR_CODE=0 4 | 5 | :init 6 | if NOT "%OS%"=="Windows_NT" goto Win9xArg 7 | if "%@eval[2+2]" == "4" goto 4NTArgs 8 | set KUKULCAN_OPTS=%* 9 | goto endInit 10 | 11 | :4NTArgs 12 | set KUKULCAN_OPTS=%$ 13 | goto endInit 14 | 15 | :Win9xArg 16 | set KUKULCAN_OPTS= 17 | 18 | :Win9xApp 19 | if %1a==a goto endInit 20 | set KUKULCAN_OPTS=%KUKULCAN_OPTS% %1 21 | shift 22 | goto Win9xApp 23 | 24 | :endInit 25 | 26 | call "%~dp0find-kukulcan-home.cmd" 27 | 28 | set /P JAVA_EXEC=<%TMP%\JAVA_EXEC 29 | set /P KUKULCAN_CLASSPATH=<%TMP%\KUKULCAN_CLASSPATH 30 | 31 | %JAVA_EXEC% %KUKULCAN_OPTS% -cp "%KUKULCAN_CLASSPATH%" com.github.mmolimar.kukulcan.repl.KukulcanRepl 32 | 33 | if ERRORLEVEL 1 goto error 34 | goto end 35 | 36 | :error 37 | if "%OS%"=="Windows_NT" @endlocal 38 | set ERROR_CODE=1 39 | 40 | :end 41 | if "%OS%"=="Windows_NT" goto endNT 42 | set JAVA_EXEC= 43 | set KUKULCAN_OPTS= 44 | set KUKULCAN_CLASSPATH= 45 | goto postExec 46 | 47 | :endNT 48 | @endlocal 49 | 50 | :postExec 51 | exit /B %ERROR_CODE% 52 | -------------------------------------------------------------------------------- /python/pykukulcan/repl/__main__.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import atexit 3 | import sys 4 | 5 | from py4j.java_gateway import JavaGateway 6 | from py4j.protocol import Py4JError 7 | 8 | from pykukulcan.repl import PyKukulcanRepl 9 | 10 | if __name__ == "__main__": 11 | parser = argparse.ArgumentParser(description="PyKukulcan REPL.") 12 | parser.add_argument("--classpath", type=str, help="Classpath for the Kukulcan libs for the Java Gateway.") 13 | options = parser.parse_args() 14 | 15 | sys.ps1 = "@ " 16 | 17 | gateway = JavaGateway.launch_gateway( 18 | classpath=options.classpath, 19 | redirect_stdout=sys.stdout, 20 | redirect_stderr=sys.stderr) 21 | 22 | 23 | def close_gateway(): 24 | gateway.shutdown() 25 | print("Bye!") 26 | 27 | 28 | atexit.register(close_gateway) 29 | 30 | try: 31 | gateway.jvm.com.github.mmolimar.kukulcan.repl.KukulcanRepl.printBanner() 32 | kukulcan = PyKukulcanRepl(gateway=gateway) 33 | except Py4JError: 34 | gateway.shutdown() 35 | sys.exit(1) 36 | -------------------------------------------------------------------------------- /kukulcan-repl/src/main/java/com/github/mmolimar/kukulcan/repl/JKukulcanRepl.java: -------------------------------------------------------------------------------- 1 | package com.github.mmolimar.kukulcan.repl; 2 | 3 | import jdk.internal.jshell.tool.JShellToolBuilder; 4 | 5 | import java.util.HashMap; 6 | import java.util.Map; 7 | import java.util.stream.Stream; 8 | 9 | /** 10 | * Entry point for the Kukulcan REPL with the JShell. 11 | * 12 | */ 13 | public class JKukulcanRepl { 14 | 15 | static String[] shellArgs(String[] args) { 16 | String[] predefs = {"--feedback", "concise"}; 17 | return Stream.of(predefs, args).flatMap(Stream::of).toArray(String[]::new); 18 | } 19 | 20 | public static void main(String[] args) throws Exception { 21 | Map prefs = new HashMap<>(); 22 | prefs.put("STARTUP", 23 | "System.out.println(com.github.mmolimar.kukulcan.repl.package$.MODULE$.banner());\n" + 24 | "import com.github.mmolimar.kukulcan.java.Kukulcan;"); 25 | 26 | JShellToolBuilder jShellToolBuilder = new JShellToolBuilder(); 27 | jShellToolBuilder.persistence(prefs); 28 | 29 | jShellToolBuilder.start(shellArgs(args)); 30 | } 31 | } 32 | -------------------------------------------------------------------------------- /project/KukulcanPackPlugin.scala: -------------------------------------------------------------------------------- 1 | import sbt.Keys._ 2 | import sbt.{Def, _} 3 | import xerial.sbt.pack._ 4 | 5 | object KukulcanPackPlugin extends AutoPlugin with PackArchive { 6 | 7 | val kukulcan = taskKey[Unit]("Creates a directory with all dependencies generated by sbt-pack plugin.") 8 | 9 | override lazy val projectSettings: Seq[Def.Setting[_]] = kukulcanSettings 10 | 11 | import PackPlugin.autoImport.{packAllUnmanagedJars, packLibJars, packModuleEntries} 12 | 13 | lazy val kukulcanSettings: Seq[Def.Setting[_]] = Seq[Def.Setting[_]]( 14 | kukulcan := { 15 | val log = streams.value.log 16 | val dirLibs = baseDirectory.value / "libs" 17 | 18 | val modules = packModuleEntries.value.map(_.file) 19 | val unmanaged = packAllUnmanagedJars.value.flatMap(_._1).map(_.data) 20 | val libs = packLibJars.value.map(_._1) 21 | 22 | dirLibs.mkdirs() 23 | IO.delete((dirLibs * "*.jar").get) 24 | val result = (modules ++ unmanaged ++ libs).distinct 25 | result.foreach(d => IO.copyFile(d, dirLibs / d.getName)) 26 | 27 | log.info(s"${result.size} dependencies needed by Kukulcan have been copied into '$dirLibs'.") 28 | } 29 | ) 30 | } 31 | -------------------------------------------------------------------------------- /kukulcan-repl/src/main/scala/com/github/mmolimar/kukulcan/repl/KukulcanRepl.scala: -------------------------------------------------------------------------------- 1 | package com.github.mmolimar.kukulcan.repl 2 | 3 | import scala.tools.nsc.{GenericRunnerSettings, Settings} 4 | import scala.tools.nsc.interpreter.ILoop 5 | 6 | /** 7 | * Entry point for the Kukulcan Scala REPL. 8 | * 9 | */ 10 | object KukulcanRepl extends App { 11 | 12 | def printBanner(): Unit = println(banner) 13 | 14 | def buildSettings: Settings = { 15 | val settings = new GenericRunnerSettings(Console.err.println) 16 | settings.usejavacp.value = true 17 | settings.processArguments(Option(args).map(_.toList).getOrElse(List.empty), processAll = true) 18 | 19 | settings 20 | } 21 | 22 | new KukulcanILoop().process(buildSettings) 23 | } 24 | 25 | private[repl] class KukulcanILoop extends ILoop { 26 | 27 | private val initCommands = 28 | """ 29 | |import com.github.mmolimar.kukulcan 30 | |""".stripMargin 31 | 32 | override def printWelcome(): Unit = KukulcanRepl.printBanner() 33 | 34 | override def prompt = "@ " 35 | 36 | override def createInterpreter(): Unit = { 37 | super.createInterpreter() 38 | intp.beQuietDuring { 39 | intp.interpret(initCommands) 40 | } 41 | } 42 | 43 | } 44 | -------------------------------------------------------------------------------- /python/setup.py: -------------------------------------------------------------------------------- 1 | try: 2 | from setuptools import setup 3 | except ImportError: 4 | from distutils.core import setup 5 | import os 6 | 7 | VERSION_PATH = os.path.join("pykukulcan", "version.py") 8 | exec(compile(open(VERSION_PATH).read(), VERSION_PATH, "exec")) 9 | VERSION = __version__ 10 | 11 | setup( 12 | name="pykukulcan", 13 | packages=["pykukulcan.repl"], 14 | description="Kukulcan for Python", 15 | version=VERSION, 16 | long_description="PyKukulcan enables Python developers to use the Kukulcan API in the Python shell " 17 | "accessing Java objects in a Java Virtual Machine.", 18 | url="https://github.com/mmolimar/kukulcan", 19 | author="Mario Molina", 20 | license="Apache License, version 2.0", 21 | classifiers=[ 22 | "Development Status :: Beta", 23 | "License :: OSI Approved :: Apache Software License", 24 | "Programming Language :: Python :: 2.7", 25 | "Programming Language :: Python :: 3", 26 | "Programming Language :: Python :: 3.4", 27 | "Programming Language :: Python :: 3.5", 28 | "Programming Language :: Python :: 3.6", 29 | "Programming Language :: Python :: 3.7" 30 | "Programming Language :: Java", 31 | "Topic :: Software Development :: REPL" 32 | ] 33 | ) 34 | -------------------------------------------------------------------------------- /kukulcan-repl/src/main/scala/com/github/mmolimar/kukulcan/repl/KApi.scala: -------------------------------------------------------------------------------- 1 | package com.github.mmolimar.kukulcan.repl 2 | 3 | import _root_.java.util.Properties 4 | 5 | import scala.reflect.io.File 6 | import scala.util.Properties.userDir 7 | 8 | class KukulcanReplException(message: String = None.orNull, cause: Throwable = None.orNull) 9 | extends RuntimeException(message, cause) 10 | 11 | private[kukulcan] abstract class KApi[K <: AnyRef](name: String) { 12 | 13 | private val KUKULCAN_ENV_VAR = "KUKULCAN_HOME" 14 | 15 | val KUKULCAN_HOME: String = sys.env.get(KUKULCAN_ENV_VAR) 16 | .orElse(sys.props.get(KUKULCAN_ENV_VAR)) 17 | .getOrElse(userDir) 18 | 19 | 20 | private var _instance: K = initialize() 21 | 22 | private def initialize(): K = { 23 | val propsFile = File(s"$KUKULCAN_HOME${File.separator}config${File.separator}${name.trim.toLowerCase}.properties") 24 | if (!propsFile.exists) { 25 | throw new KukulcanReplException(s"Cannot locate config file '${propsFile.jfile.getAbsolutePath}'. " + 26 | s"Do you have properly set '$KUKULCAN_ENV_VAR' environment variable?") 27 | } 28 | val props = new Properties() 29 | props.load(propsFile.inputStream()) 30 | createInstance(props) 31 | } 32 | 33 | final def reload(): Unit = _instance = initialize() 34 | 35 | private[kukulcan] def inst: K = _instance 36 | 37 | protected def createInstance(props: Properties): K 38 | 39 | } 40 | -------------------------------------------------------------------------------- /bin/jkukulcan.cmd: -------------------------------------------------------------------------------- 1 | @echo off 2 | 3 | set ERROR_CODE=0 4 | 5 | :init 6 | set KUKULCAN_OPTS="--add-exports=jdk.jshell/jdk.internal.jshell.tool=ALL-UNNAMED" 7 | 8 | if NOT "%OS%"=="Windows_NT" goto Win9xArg 9 | if "%@eval[2+2]" == "4" goto 4NTArgs 10 | set KUKULCAN_OPTS=%KUKULCAN_OPTS% %* 11 | goto endInit 12 | 13 | :4NTArgs 14 | set KUKULCAN_OPTS=%KUKULCAN_OPTS% %$ 15 | goto endInit 16 | 17 | :Win9xArg 18 | set KUKULCAN_OPTS=%KUKULCAN_OPTS% 19 | 20 | :Win9xApp 21 | if %1a==a goto endInit 22 | set KUKULCAN_OPTS=%KUKULCAN_OPTS% %1 23 | shift 24 | goto Win9xApp 25 | 26 | :endInit 27 | 28 | call "%~dp0find-kukulcan-home.cmd" 29 | 30 | set /P JAVA_EXEC=<%TMP%\JAVA_EXEC 31 | set /P KUKULCAN_CLASSPATH=<%TMP%\KUKULCAN_CLASSPATH 32 | 33 | set KUKULCAN_FULL_CLASSPATH= 34 | for %%i in ("%KUKULCAN_CLASSPATH%") do ( 35 | call :concat "%%i" 36 | ) 37 | 38 | %JAVA_EXEC% %KUKULCAN_OPTS% -cp "%KUKULCAN_CLASSPATH%" com.github.mmolimar.kukulcan.repl.JKukulcanRepl --class-path "%KUKULCAN_FULL_CLASSPATH%" 39 | 40 | if ERRORLEVEL 1 goto error 41 | goto end 42 | 43 | :error 44 | if "%OS%"=="Windows_NT" @endlocal 45 | set ERROR_CODE=1 46 | 47 | :end 48 | if "%OS%"=="Windows_NT" goto endNT 49 | set JAVA_EXEC= 50 | set KUKULCAN_OPTS= 51 | set KUKULCAN_CLASSPATH= 52 | set KUKULCAN_FULL_CLASSPATH= 53 | goto postExec 54 | 55 | :endNT 56 | @endlocal 57 | 58 | :postExec 59 | exit /B %ERROR_CODE% 60 | 61 | :concat 62 | if not defined KUKULCAN_FULL_CLASSPATH ( 63 | set KUKULCAN_FULL_CLASSPATH=%~1 64 | ) else ( 65 | set KUKULCAN_FULL_CLASSPATH=%KUKULCAN_FULL_CLASSPATH%;%~1 66 | ) 67 | -------------------------------------------------------------------------------- /config/connect.properties: -------------------------------------------------------------------------------- 1 | ######################################################################################################################## 2 | # # 3 | # Properties for the Connect API # 4 | # # 5 | ######################################################################################################################## 6 | 7 | # Kafka-Connect host. 8 | rest.host.name=localhost 9 | 10 | # Kafka-Connect port. 11 | rest.port=8083 12 | 13 | # Time in seconds before a request times out. 14 | request.timeout.seconds=300 15 | 16 | # User to use when authenticating requests to Kafka-Connect. 17 | #rest.auth.basic.username= 18 | 19 | # Password to use when authenticating requests to Kafka-Connect. 20 | #rest.auth.basic.password= 21 | 22 | # Proxy host. 23 | #proxy.host= 24 | 25 | # Proxy port. 26 | #proxy.port= 27 | 28 | # Scheme to use for the proxy. Allowed values: 'HTTP' or 'HTTPS'. 29 | #proxy.scheme= 30 | 31 | # User to use in the proxy. 32 | #proxy.auth.username= 33 | 34 | # Password to use in the proxy. 35 | #proxy.auth.password= 36 | 37 | # File path to keystore. 38 | #ssl.keystore.location= 39 | 40 | # Password for keystore (do not fill if there is no password). 41 | #ssl.keystore.password= 42 | 43 | # File path to truststore. 44 | #ssl.truststore.location= 45 | 46 | # Password for truststore (do not fill if there is no password). 47 | #ssl.truststore.password= 48 | 49 | # Flag to skip all validation of SSL Certificates. 50 | #ssl.certificates.insecure= 51 | -------------------------------------------------------------------------------- /bin/find-kukulcan-home: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | cygwin=false 4 | mingw=false 5 | darwin=false 6 | case "`uname`" in 7 | CYGWIN*) cygwin=true 8 | ;; 9 | MINGW*) mingw=true 10 | ;; 11 | Darwin*) darwin=true 12 | if [ -z "$JAVA_VERSION" ] ; then 13 | JAVA_VERSION="CurrentJDK" 14 | else 15 | echo "Using Java version: $JAVA_VERSION" 1>&2 16 | fi 17 | if [ -z "$JAVA_HOME" ] ; then 18 | JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/${JAVA_VERSION}/Home 19 | fi 20 | ;; 21 | esac 22 | 23 | if $cygwin; then 24 | [ -n "$JAVA_HOME" ] && 25 | JAVA_HOME=`cygpath -am "$JAVA_HOME"` 26 | fi 27 | 28 | if $mingw ; then 29 | [ -n "$JAVA_HOME" ] && 30 | JAVA_HOME="`(cd "$JAVA_HOME"; pwd -W | sed 's|/|\\\\|g')`" 31 | fi 32 | 33 | if [ -n "${JAVA_HOME}" ]; then 34 | export JAVA_EXEC="${JAVA_HOME}/bin/java" 35 | else 36 | if [ "$(command -v java)" ]; then 37 | export JAVA_EXEC="java" 38 | else 39 | echo "JAVA_HOME is not set." >&2 40 | exit 1 41 | fi 42 | fi 43 | 44 | if [ -z "${KUKULCAN_HOME}" ]; then 45 | CURRENT="$0" 46 | while [ -h "$CURRENT" ] ; do 47 | ls=`ls -ld "$CURRENT"` 48 | link=`expr "$ls" : '.*-> \(.*\)$'` 49 | if expr "$link" : '/.*' > /dev/null; then 50 | CURRENT="$link" 51 | else 52 | CURRENT="`dirname "$CURRENT"`/$link" 53 | fi 54 | done 55 | saveddir=`pwd` 56 | KUKULCAN_HOME=`dirname "$CURRENT"`/.. 57 | KUKULCAN_HOME=`cd "$KUKULCAN_HOME" && pwd` 58 | cd "$saveddir" 59 | fi 60 | 61 | KUKULCAN_LIBS_DIR="${KUKULCAN_HOME}/libs" 62 | if [ ! -d "$KUKULCAN_LIBS_DIR" ] || [ -z "$(ls -A -- "$KUKULCAN_LIBS_DIR")" ]; then 63 | echo "Libs directory '$KUKULCAN_LIBS_DIR' does not exist or is empty." 64 | echo "Execute the command 'sbt kukulcan' to create the libs directory." 65 | exit 1 66 | fi 67 | 68 | export KUKULCAN_CLASSPATH="$KUKULCAN_LIBS_DIR/*" 69 | -------------------------------------------------------------------------------- /kukulcan-repl/src/main/scala/com/github/mmolimar/kukulcan/repl/package.scala: -------------------------------------------------------------------------------- 1 | package com.github.mmolimar.kukulcan 2 | 3 | package object repl { 4 | 5 | val banner: String = 6 | """ 7 | | ,/`. 8 | | ---_ ......._-_--. ,'/ __`. 9 | | (|\ / / /| \ \ ,'_/_ _ _`. 10 | | / / .' -=-' `. ,'__/_ ___ __ `. 11 | | / / .' ) ,'_ /___ __ _ __ `. 12 | | _ __ _ _ _/ / .' _.) / '-.._/______-"-______`. 13 | || | / / | | | | / o o _.-' / .' 14 | || |/ / _ _ | | __ _ _ | | ___ __ _ _ __ \ _.-' / .'*| 15 | || \ | | | || |/ /| | | || | / __| / _` || '_ \ \______.-'// .'.' \*| 16 | || |\ \| |_| || < | |_| || || (__ | (_| || | | | \| \ | // .'.' _ |*| 17 | |\_| \_/ \__,_||_|\_\ \__,_||_| \___| \__,_||_| |_| ` \|// .'.'_ _ _|*| 18 | | . .// .'.' | _ _ \*| 19 | | \`-|\_/ / \ _ _ \*\ 20 | | A REPL for Apache Kafka `/'\__/ \ _ _ \*\ 21 | | /^| \ _ _ \* 22 | | \_ """ 23 | .stripMargin 24 | .concat( 25 | s"""Version: ${BuildInfo.version} 26 | |----------------------------------------------------------------------------------------------------------- 27 | | 28 | |""".stripMargin 29 | ) 30 | } 31 | -------------------------------------------------------------------------------- /kukulcan-api/src/test/scala/com/github/mmolimar/kukulcan/KAdminMetricsSpec.scala: -------------------------------------------------------------------------------- 1 | package com.github.mmolimar.kukulcan 2 | 3 | import _root_.com.github.mmolimar.kukulcan.{KAdmin => SKAdmin} 4 | 5 | import com.github.mmolimar.kukulcan.java.{KAdmin => JKAdmin} 6 | import net.manub.embeddedkafka.{EmbeddedKafka, EmbeddedKafkaConfig} 7 | import org.apache.kafka.clients.admin.KafkaAdminClient 8 | 9 | import _root_.java.util.Properties 10 | 11 | class KAdminMetricsSpec extends KukulcanApiTestHarness with EmbeddedKafka { 12 | 13 | lazy implicit val config: EmbeddedKafkaConfig = EmbeddedKafkaConfig() 14 | 15 | override def apiClass: Class[_] = classOf[SKAdmin] 16 | 17 | override def execScalaTests(): Unit = withRunningKafka { 18 | val scalaApi: SKAdmin = { 19 | val props = new Properties() 20 | props.put("bootstrap.servers", s"localhost:${config.kafkaPort}") 21 | SKAdmin(props) 22 | } 23 | 24 | scalaApi.servers shouldBe s"localhost:${config.kafkaPort}" 25 | scalaApi.client.isInstanceOf[KafkaAdminClient] shouldBe true 26 | 27 | scalaApi.metrics.getMetrics.size shouldNot be(0) 28 | scalaApi.metrics.getMetrics(".*", ".*").size shouldNot be(0) 29 | scalaApi.metrics.getMetrics("app-info", "version").head._2.metricValue() shouldBe "2.7.0" 30 | 31 | scalaApi.metrics.listMetrics() 32 | scalaApi.metrics.listMetrics("app-info", "version") 33 | } 34 | 35 | override def execJavaTests(): Unit = withRunningKafka { 36 | import scala.collection.JavaConverters._ 37 | 38 | val javaApi: JKAdmin = { 39 | val props = new Properties() 40 | props.put("bootstrap.servers", s"localhost:${config.kafkaPort}") 41 | new JKAdmin(props) 42 | } 43 | 44 | javaApi.servers shouldBe s"localhost:${config.kafkaPort}" 45 | javaApi.client.isInstanceOf[KafkaAdminClient] shouldBe true 46 | 47 | javaApi.metrics.getMetrics.size shouldNot be(0) 48 | javaApi.metrics.getMetrics(".*", ".*").size shouldNot be(0) 49 | javaApi.metrics.getMetrics("app-info", "version").asScala.head._2.metricValue() shouldBe "2.7.0" 50 | 51 | javaApi.metrics.listMetrics() 52 | javaApi.metrics.listMetrics("app-info", "version") 53 | } 54 | 55 | } 56 | -------------------------------------------------------------------------------- /kukulcan-api/src/main/scala/com/github/mmolimar/kukulcan/KProducer.scala: -------------------------------------------------------------------------------- 1 | package com.github.mmolimar.kukulcan 2 | 3 | import _root_.java.util.Properties 4 | 5 | import org.apache.kafka.clients.producer.KafkaProducer 6 | import org.apache.kafka.tools.{ToolsUtils => JToolsUtils} 7 | 8 | import scala.collection.JavaConverters._ 9 | 10 | /** 11 | * Factory for [[com.github.mmolimar.kukulcan.KProducer]] instances. 12 | * 13 | */ 14 | object KProducer { 15 | 16 | def apply[K, V](props: Properties): KProducer[K, V] = new KProducer(props) 17 | 18 | } 19 | 20 | /** 21 | * An enriched implementation of the {@code org.apache.kafka.clients.producer.KafkaProducer} class to 22 | * produce messages in Kafka. 23 | * 24 | * @param props Properties with the configuration. 25 | */ 26 | class KProducer[K, V](val props: Properties) extends KafkaProducer[K, V](props) { 27 | 28 | import org.apache.kafka.common.{Metric, MetricName} 29 | 30 | /** 31 | * Get all metrics registered. 32 | * 33 | * @return a {@code Map[MetricName, Metric]} with the all metrics registered. 34 | */ 35 | def getMetrics: Map[MetricName, Metric] = { 36 | getMetrics(".*", ".*") 37 | } 38 | 39 | /** 40 | * Get all metrics registered filtered by the group and name regular expressions. 41 | * 42 | * @param groupRegex Regex to filter metrics by group name. 43 | * @param nameRegex Regex to filter metrics by its name. 44 | * @return a {@code Map[MetricName, Metric]} with the all metrics registered filtered by the group and name regular 45 | * expressions. 46 | */ 47 | def getMetrics(groupRegex: String, nameRegex: String): Map[MetricName, Metric] = { 48 | metrics.asScala 49 | .filter(metric => metric._1.group.matches(groupRegex) && metric._1.name.matches(nameRegex)) 50 | .toMap 51 | } 52 | 53 | /** 54 | * Print all metrics. 55 | */ 56 | def listMetrics(): Unit = { 57 | listMetrics(".*", ".*") 58 | } 59 | 60 | /** 61 | * Print all metrics filtered by the group and name regular expressions. 62 | * 63 | * @param groupRegex Regex to filter metrics by group name. 64 | * @param nameRegex Regex to filter metrics by its name. 65 | */ 66 | def listMetrics(groupRegex: String, nameRegex: String): Unit = { 67 | JToolsUtils.printMetrics(getMetrics(groupRegex, nameRegex).asJava) 68 | } 69 | 70 | } 71 | -------------------------------------------------------------------------------- /kukulcan-api/src/test/scala/com/github/mmolimar/kukulcan/KProducerSpec.scala: -------------------------------------------------------------------------------- 1 | package com.github.mmolimar.kukulcan 2 | 3 | import _root_.com.github.mmolimar.kukulcan.java.{KProducer => JKProducer} 4 | import _root_.com.github.mmolimar.kukulcan.{KProducer => SKProducer} 5 | 6 | import net.manub.embeddedkafka.{EmbeddedKafka, EmbeddedKafkaConfig} 7 | import org.apache.kafka.clients.producer.KafkaProducer 8 | 9 | import _root_.java.util.Properties 10 | 11 | class KProducerSpec extends KukulcanApiTestHarness with EmbeddedKafka { 12 | 13 | lazy implicit val config: EmbeddedKafkaConfig = EmbeddedKafkaConfig() 14 | 15 | override def apiClass: Class[_] = classOf[SKProducer[String, String]] 16 | 17 | override def execScalaTests(): Unit = withRunningKafka { 18 | val scalaApi: SKProducer[String, String] = { 19 | val props = new Properties() 20 | props.put("bootstrap.servers", s"localhost:${config.kafkaPort}") 21 | props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer") 22 | props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer") 23 | SKProducer[String, String](props) 24 | } 25 | 26 | scalaApi.isInstanceOf[KafkaProducer[String, String]] shouldBe true 27 | 28 | scalaApi.getMetrics shouldBe scalaApi.getMetrics(".*", ".*") 29 | scalaApi.getMetrics("app-info", "version").head._2.metricValue() shouldBe "2.7.0" 30 | 31 | scalaApi.listMetrics() 32 | scalaApi.listMetrics("app-info", "version") 33 | } 34 | 35 | override def execJavaTests(): Unit = withRunningKafka { 36 | val javaApi: JKProducer[String, String] = { 37 | val props = new Properties() 38 | props.put("bootstrap.servers", s"localhost:${config.kafkaPort}") 39 | props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer") 40 | props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer") 41 | new JKProducer[String, String](props) 42 | } 43 | 44 | javaApi.isInstanceOf[KafkaProducer[String, String]] shouldBe true 45 | 46 | javaApi.getMetrics shouldBe javaApi.getMetrics(".*", ".*") 47 | javaApi.getMetrics("app-info", "version").values().iterator().next().metricValue() shouldBe "2.7.0" 48 | 49 | javaApi.listMetrics() 50 | javaApi.listMetrics("app-info", "version") 51 | } 52 | 53 | } 54 | -------------------------------------------------------------------------------- /kukulcan-api/src/main/java/com/github/mmolimar/kukulcan/java/KUtils.java: -------------------------------------------------------------------------------- 1 | package com.github.mmolimar.kukulcan.java; 2 | 3 | import scala.Predef; 4 | import scala.collection.JavaConverters; 5 | 6 | import java.util.List; 7 | import java.util.Map; 8 | import java.util.Optional; 9 | 10 | /** 11 | * Utility class to convert class from Java to Scala and viceversa. 12 | */ 13 | public class KUtils { 14 | 15 | /** 16 | * Convert a Java list to a Scala immutable Seq. 17 | * 18 | * @param list The Java {@code List[K]} to convert. 19 | * @return the immutable Scala {@code Seq[K]} converted. 20 | */ 21 | public static scala.collection.immutable.Seq toScalaSeq(List list) { 22 | return JavaConverters.asScalaBufferConverter(list).asScala().toList(); 23 | } 24 | 25 | /** 26 | * Convert a Java map to a Scala immutable Map. 27 | * 28 | * @param map The Java {@code Map[K, V]} to convert. 29 | * @return the immutable Scala {@code Map[K, V]} converted. 30 | */ 31 | public static scala.collection.immutable.Map toScalaMap(Map map) { 32 | return JavaConverters.mapAsScalaMap(map).toMap(Predef.$conforms()); 33 | } 34 | 35 | /** 36 | * Convert a Scala seq to a Java list. 37 | * 38 | * @param seq The Scala {@code Seq[K]} to convert. 39 | * @return the Java {@code List[K]} converted. 40 | */ 41 | public static List toJavaList(scala.collection.Seq seq) { 42 | return JavaConverters.seqAsJavaList(seq); 43 | } 44 | 45 | /** 46 | * Convert a Scala map to a Java map. 47 | * 48 | * @param map The Scala {@code Map[K, V]} to convert. 49 | * @return the Java {@code Map[K, V]} converted. 50 | */ 51 | public static Map toJavaMap(scala.collection.Map map) { 52 | return JavaConverters.mapAsJavaMap(map); 53 | } 54 | 55 | /** 56 | * Convert a Scala option to a Java optional. 57 | * 58 | * @param opt The Scala {@code Option[K]} to convert. 59 | * @return the Java {@code Optional[K]} converted. 60 | */ 61 | public static Optional toJavaOption(scala.Option opt) { 62 | return Optional.ofNullable(opt.getOrElse(() -> null)); 63 | } 64 | 65 | /** 66 | * Create a Scala option for an object. 67 | * 68 | * @param obj The object to create its Scala option. 69 | * @return the Scala {@code Option[K]}. 70 | */ 71 | public static scala.Option scalaOption(K obj) { 72 | return scala.Option.apply(obj); 73 | } 74 | 75 | } 76 | -------------------------------------------------------------------------------- /kukulcan-api/src/main/scala/com/github/mmolimar/kukulcan/KConsumer.scala: -------------------------------------------------------------------------------- 1 | package com.github.mmolimar.kukulcan 2 | 3 | import _root_.java.util.Properties 4 | 5 | import org.apache.kafka.clients.consumer.KafkaConsumer 6 | import org.apache.kafka.tools.{ToolsUtils => JToolsUtils} 7 | 8 | import scala.collection.JavaConverters._ 9 | 10 | /** 11 | * Factory for [[com.github.mmolimar.kukulcan.KConsumer]] instances. 12 | * 13 | */ 14 | object KConsumer { 15 | 16 | def apply[K, V](props: Properties): KConsumer[K, V] = new KConsumer(props) 17 | 18 | } 19 | 20 | /** 21 | * An enriched implementation of the {@code org.apache.kafka.clients.consumer.KafkaConsumer} class to 22 | * consume messages in Kafka. 23 | * 24 | * @param props Properties with the configuration. 25 | */ 26 | class KConsumer[K, V](val props: Properties) extends KafkaConsumer[K, V](props) { 27 | 28 | import org.apache.kafka.common.{Metric, MetricName} 29 | 30 | /** 31 | * Subscribe to a topic 32 | * 33 | * @param topic Topic name. 34 | */ 35 | def subscribe(topic: String): Unit = subscribe(Seq(topic)) 36 | 37 | /** 38 | * Subscribe to a topic list. 39 | * 40 | * @param topics Topic list. 41 | */ 42 | def subscribe(topics: Seq[String]): Unit = subscribe(topics.asJava) 43 | 44 | /** 45 | * Get all metrics registered. 46 | * 47 | * @return a {@code Map[MetricName, Metric]} with the all metrics registered. 48 | */ 49 | def getMetrics: Map[MetricName, Metric] = { 50 | getMetrics(".*", ".*") 51 | } 52 | 53 | /** 54 | * Get all metrics registered filtered by the group and name regular expressions. 55 | * 56 | * @param groupRegex Regex to filter metrics by group name. 57 | * @param nameRegex Regex to filter metrics by its name. 58 | * @return a {@code Map[MetricName, Metric]} with the all metrics registered filtered by the group and name regular 59 | * expressions. 60 | */ 61 | def getMetrics(groupRegex: String, nameRegex: String): Map[MetricName, Metric] = { 62 | metrics.asScala 63 | .filter(metric => metric._1.group.matches(groupRegex) && metric._1.name.matches(nameRegex)) 64 | .toMap 65 | } 66 | 67 | /** 68 | * Print all metrics. 69 | */ 70 | def listMetrics(): Unit = { 71 | listMetrics(".*", ".*") 72 | } 73 | 74 | /** 75 | * Print all metrics filtered by the group and name regular expressions. 76 | * 77 | * @param groupRegex Regex to filter metrics by group name. 78 | * @param nameRegex Regex to filter metrics by its name. 79 | */ 80 | def listMetrics(groupRegex: String, nameRegex: String): Unit = { 81 | JToolsUtils.printMetrics(getMetrics(groupRegex, nameRegex).asJava) 82 | } 83 | 84 | } 85 | -------------------------------------------------------------------------------- /kukulcan-api/src/test/scala/com/github/mmolimar/kukulcan/KConsumerSpec.scala: -------------------------------------------------------------------------------- 1 | package com.github.mmolimar.kukulcan 2 | 3 | import _root_.com.github.mmolimar.kukulcan.java.{KConsumer => JKConsumer} 4 | import _root_.com.github.mmolimar.kukulcan.{KConsumer => SKConsumer} 5 | 6 | import net.manub.embeddedkafka.{EmbeddedKafka, EmbeddedKafkaConfig} 7 | import org.apache.kafka.clients.consumer.KafkaConsumer 8 | 9 | import _root_.java.util.Properties 10 | 11 | class KConsumerSpec extends KukulcanApiTestHarness with EmbeddedKafka { 12 | 13 | lazy implicit val config: EmbeddedKafkaConfig = EmbeddedKafkaConfig() 14 | 15 | override def apiClass: Class[_] = classOf[SKConsumer[String, String]] 16 | 17 | override def execScalaTests(): Unit = withRunningKafka { 18 | val scalaApi: SKConsumer[String, String] = { 19 | val props = new Properties() 20 | props.put("bootstrap.servers", s"localhost:${config.kafkaPort}") 21 | props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer") 22 | props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer") 23 | props.put("group.id", "test-group") 24 | SKConsumer[String, String](props) 25 | } 26 | val kadmin = { 27 | val props = new Properties() 28 | props.put("bootstrap.servers", s"localhost:${config.kafkaPort}") 29 | KAdmin(props) 30 | } 31 | kadmin.topics.createTopic("test", 1, 1) 32 | 33 | scalaApi.isInstanceOf[KafkaConsumer[String, String]] shouldBe true 34 | 35 | scalaApi.getMetrics shouldBe scalaApi.getMetrics(".*", ".*") 36 | scalaApi.getMetrics("app-info", "version").head._2.metricValue() shouldBe "2.7.0" 37 | 38 | scalaApi.listMetrics() 39 | scalaApi.listMetrics("app-info", "version") 40 | 41 | scalaApi.subscribe("test") 42 | } 43 | 44 | override def execJavaTests(): Unit = withRunningKafka { 45 | val javaApi: JKConsumer[String, String] = { 46 | val props = new Properties() 47 | props.put("bootstrap.servers", s"localhost:${config.kafkaPort}") 48 | props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer") 49 | props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer") 50 | props.put("group.id", "test-group") 51 | new JKConsumer[String, String](props) 52 | } 53 | val kadmin = { 54 | val props = new Properties() 55 | props.put("bootstrap.servers", s"localhost:${config.kafkaPort}") 56 | KAdmin(props) 57 | } 58 | kadmin.topics.createTopic("test", 1, 1) 59 | 60 | javaApi.isInstanceOf[KafkaConsumer[String, String]] shouldBe true 61 | 62 | javaApi.getMetrics shouldBe javaApi.getMetrics(".*", ".*") 63 | javaApi.getMetrics("app-info", "version").values().iterator().next().metricValue() shouldBe "2.7.0" 64 | 65 | javaApi.listMetrics() 66 | javaApi.listMetrics("app-info", "version") 67 | 68 | javaApi.subscribe("test") 69 | } 70 | 71 | } 72 | -------------------------------------------------------------------------------- /python/pykukulcan/repl/__init__.py: -------------------------------------------------------------------------------- 1 | __all__ = ['PyKukulcanRepl'] 2 | 3 | 4 | class PyKukulcanRepl: 5 | """ 6 | Entry point for the PyKukulcan REPL. 7 | """ 8 | 9 | _gateway = None 10 | 11 | def __init__(self, gateway): 12 | self._gateway = gateway 13 | 14 | def admin(self): 15 | """ 16 | Create a KAdmin instance reading the {@code admin.properties} file. 17 | If the instance was already created, it will be reused. 18 | 19 | :return: The KAdmin instance initialized. 20 | """ 21 | return self._gateway.jvm.com.github.mmolimar.kukulcan.java.Kukulcan.admin() 22 | 23 | def consumer(self): 24 | """ 25 | Create a KConsumer instance reading the {@code consumer.properties} file. 26 | If the instance was already created, it will be reused. 27 | 28 | :return: The KConsumer instance initialized. 29 | """ 30 | return self._gateway.jvm.com.github.mmolimar.kukulcan.java.Kukulcan.consumer() 31 | 32 | def producer(self): 33 | """ 34 | Create a KProducer instance reading the {@code producer.properties} file. 35 | If the instance was already created, it will be reused. 36 | 37 | :return: The KProducer instance initialized. 38 | """ 39 | return self._gateway.jvm.com.github.mmolimar.kukulcan.java.Kukulcan.producer() 40 | 41 | def connect(self): 42 | """ 43 | Create a KConnect instance reading the {@code connect.properties} file. 44 | If the instance was already created, it will be reused. 45 | 46 | :return: The KConnect instance initialized. 47 | """ 48 | return self._gateway.jvm.com.github.mmolimar.kukulcan.java.Kukulcan.connect() 49 | 50 | def streams(self, topology): 51 | """ 52 | Create a KStreams instance reading the {@code streams.properties} file. 53 | If the instance was already created, it will be reused. 54 | 55 | :return: The KStreams instance initialized. 56 | """ 57 | return self._gateway.jvm.com.github.mmolimar.kukulcan.java.Kukulcan.streams(topology) 58 | 59 | def schema_registry(self): 60 | """ 61 | Create a KSchemaRegistry instance reading the {@code schema-registry.properties} file. 62 | If the instance was already created, it will be reused. 63 | 64 | :return: The KSchemaRegistry instance initialized. 65 | """ 66 | return self._gateway.jvm.com.github.mmolimar.kukulcan.java.Kukulcan.schemaRegistry() 67 | 68 | def ksql(self): 69 | """ 70 | Create a KKsql instance reading the {@code ksql.properties} file. 71 | If the instance was already created, it will be reused. 72 | 73 | :return: The KKsql instance initialized. 74 | """ 75 | return self._gateway.jvm.com.github.mmolimar.kukulcan.java.Kukulcan.ksql() 76 | 77 | def reload(self): 78 | """ 79 | Re-create all instances using their properties files. 80 | """ 81 | self._gateway.jvm.com.github.mmolimar.kukulcan.java.Kukulcan.reload() 82 | -------------------------------------------------------------------------------- /kukulcan-api/src/main/java/com/github/mmolimar/kukulcan/java/KProducer.java: -------------------------------------------------------------------------------- 1 | package com.github.mmolimar.kukulcan.java; 2 | 3 | import org.apache.kafka.clients.producer.KafkaProducer; 4 | import org.apache.kafka.common.Metric; 5 | import org.apache.kafka.common.MetricName; 6 | import org.apache.kafka.tools.ToolsUtils; 7 | 8 | import java.util.Map; 9 | import java.util.Properties; 10 | import java.util.stream.Collectors; 11 | 12 | /** 13 | * An enriched implementation of the {@code org.apache.kafka.clients.producer.KafkaProducer} class to 14 | * produce messages in Kafka. 15 | */ 16 | public class KProducer extends KafkaProducer { 17 | 18 | /** 19 | * @param props Properties with the configuration. 20 | */ 21 | public KProducer(Properties props) { 22 | super(props); 23 | } 24 | 25 | /** 26 | * Get all metrics registered. 27 | * 28 | * @return a {@code Map[MetricName, Metric]} with the all metrics registered. 29 | */ 30 | public Map getMetrics() { 31 | return getMetrics(".*"); 32 | } 33 | 34 | /** 35 | * Get all metrics registered filtered by the group regular expressions. 36 | * 37 | * @param groupRegex Regex to filter metrics by group name. 38 | * @return a {@code Map[MetricName, Metric]} with the all metrics registered filtered by the group regular 39 | * expression. 40 | */ 41 | public Map getMetrics(String groupRegex) { 42 | return getMetrics(groupRegex, ".*"); 43 | } 44 | 45 | /** 46 | * Get all metrics registered filtered by the group and name regular expressions. 47 | * 48 | * @param groupRegex Regex to filter metrics by group name. 49 | * @param nameRegex Regex to filter metrics by its name. 50 | * @return a {@code Map[MetricName, Metric]} with the all metrics registered filtered by the group and name 51 | * regular expressions. 52 | */ 53 | public Map getMetrics(String groupRegex, String nameRegex) { 54 | return metrics().entrySet().stream() 55 | .filter(metric -> metric.getKey().group().matches(groupRegex) && 56 | metric.getKey().name().matches(nameRegex)) 57 | .collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue)); 58 | } 59 | 60 | /** 61 | * Print all metrics. 62 | */ 63 | public void listMetrics() { 64 | listMetrics(".*"); 65 | } 66 | 67 | /** 68 | * Print all metrics filtered by the group regular expression. 69 | * 70 | * @param groupRegex Regex to filter metrics by group name. 71 | */ 72 | public void listMetrics(String groupRegex) { 73 | listMetrics(groupRegex, ".*"); 74 | } 75 | 76 | /** 77 | * Print all metrics filtered by the group and name regular expressions. 78 | * 79 | * @param groupRegex Regex to filter metrics by group name. 80 | * @param nameRegex Regex to filter metrics by its name. 81 | */ 82 | public void listMetrics(String groupRegex, String nameRegex) { 83 | ToolsUtils.printMetrics(getMetrics(groupRegex, nameRegex)); 84 | } 85 | } 86 | -------------------------------------------------------------------------------- /config/schema-registry.properties: -------------------------------------------------------------------------------- 1 | ######################################################################################################################## 2 | # # 3 | # Properties for the SchemaRegistry client API # 4 | # # 5 | ######################################################################################################################## 6 | 7 | # Comma-separated list of URLs for schema registry instances that can be used to register or look up schemas. If you wish to get a connection to a mocked schema registry for testing, you can specify a scope using the 'mock://' pseudo-protocol. For example, 'mock://my-scope-name'. 8 | schema.registry.url=http://0.0.0.0:8081 9 | 10 | # Maximum number of schemas to create or cache locally. 11 | #max.schemas.per.subject=1000 12 | 13 | # The SSL protocol used to generate the SSLContext. The default is 'TLSv1.3' when running with Java 11 or newer, 'TLSv1.2' otherwise. This value should be fine for most use cases. Allowed values in recent JVMs are 'TLSv1.2' and 'TLSv1.3'. 'TLS', 'TLSv1.1', 'SSL', 'SSLv2' and 'SSLv3' may be supported in older JVMs, but their usage is discouraged due to known security vulnerabilities. With the default value for this config and 'ssl.enabled.protocols', clients will downgrade to 'TLSv1.2' if the server does not support 'TLSv1.3'. If this config is set to 'TLSv1.2', clients will not use 'TLSv1.3' even if it is one of the values in ssl.enabled.protocols and the server only supports 'TLSv1.3'. 14 | #schema.registry.ssl.protocol= 15 | 16 | # The name of the security provider used for SSL connections. Default value is the default security provider of the JVM. 17 | #schema.registry.ssl.provider= 18 | 19 | # The algorithm used by key manager factory for SSL connections. Default value is the key manager factory algorithm configured for the Java Virtual Machine. 20 | #schema.registry.ssl.keymanager.algorithm= 21 | 22 | # The algorithm used by the trust manager factory for SSL connections. Default value is the trust manager factory algorithm configured for the Java Virtual Machine. 23 | #schema.registry.ssl.trustmanager.algorithm= 24 | 25 | # The file format of the key store file. This is optional for client. 26 | #schema.registry.ssl.keystore.type=JKS 27 | 28 | # The location of the key store file. This is optional for client and can be used for two-way authentication for client. 29 | #schema.registry.keystore.location= 30 | 31 | # The store password for the key store file. This is optional for client and only needed if ssl.keystore.location is configured. 32 | #schema.registry.keystore.password= 33 | 34 | # The password of the private key in the key store file. This is optional for client. 35 | #schema.registry.ssl.key.password= 36 | 37 | # The file format of the trust store file. 38 | #schema.registry.ssl.truststore.type=JKS 39 | 40 | # The location of the trust store file. 41 | #schema.registry.ssl.truststore.location= 42 | 43 | # The password for the trust store file. If a password is not set access to the truststore is still available, but integrity checking is disabled. 44 | #schema.registry.ssl.truststore.password= 45 | -------------------------------------------------------------------------------- /kukulcan-api/src/main/java/com/github/mmolimar/kukulcan/java/KConsumer.java: -------------------------------------------------------------------------------- 1 | package com.github.mmolimar.kukulcan.java; 2 | 3 | import org.apache.kafka.clients.consumer.KafkaConsumer; 4 | import org.apache.kafka.common.Metric; 5 | import org.apache.kafka.common.MetricName; 6 | import org.apache.kafka.tools.ToolsUtils; 7 | 8 | import java.util.Collections; 9 | import java.util.Map; 10 | import java.util.Properties; 11 | import java.util.stream.Collectors; 12 | 13 | /** 14 | * An enriched implementation of the {@code org.apache.kafka.clients.consumer.KafkaConsumer} class to 15 | * consume messages in Kafka. 16 | */ 17 | public class KConsumer extends KafkaConsumer { 18 | 19 | /** 20 | * @param props Properties with the configuration. 21 | */ 22 | public KConsumer(Properties props) { 23 | super(props); 24 | } 25 | 26 | /** 27 | * Subscribe to a topic 28 | * 29 | * @param topic Topic name. 30 | */ 31 | public void subscribe(String topic) { 32 | subscribe(Collections.singletonList(topic)); 33 | } 34 | 35 | /** 36 | * Get all metrics registered. 37 | * 38 | * @return a {@code Map[MetricName, Metric]} with the all metrics registered. 39 | */ 40 | public Map getMetrics() { 41 | return getMetrics(".*"); 42 | } 43 | 44 | /** 45 | * Get all metrics registered filtered by the group regular expressions. 46 | * 47 | * @param groupRegex Regex to filter metrics by group name. 48 | * @return a {@code Map[MetricName, Metric]} with the all metrics registered filtered by the group regular 49 | * expression. 50 | */ 51 | public Map getMetrics(String groupRegex) { 52 | return getMetrics(groupRegex, ".*"); 53 | } 54 | 55 | /** 56 | * Get all metrics registered filtered by the group and name regular expressions. 57 | * 58 | * @param groupRegex Regex to filter metrics by group name. 59 | * @param nameRegex Regex to filter metrics by its name. 60 | * @return a {@code Map[MetricName, Metric]} with the all metrics registered filtered by the group and name regular 61 | * expressions. 62 | */ 63 | public Map getMetrics(String groupRegex, String nameRegex) { 64 | return metrics().entrySet().stream() 65 | .filter(metric -> metric.getKey().group().matches(groupRegex) && 66 | metric.getKey().name().matches(nameRegex)) 67 | .collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue)); 68 | } 69 | 70 | /** 71 | * Print all metrics. 72 | */ 73 | public void listMetrics() { 74 | listMetrics(".*"); 75 | } 76 | 77 | /** 78 | * Print all metrics filtered by the group regular expression. 79 | * 80 | * @param groupRegex Regex to filter metrics by group name. 81 | */ 82 | public void listMetrics(String groupRegex) { 83 | listMetrics(groupRegex, ".*"); 84 | } 85 | 86 | /** 87 | * Print all metrics filtered by the group and name regular expressions. 88 | * 89 | * @param groupRegex Regex to filter metrics by group name. 90 | * @param nameRegex Regex to filter metrics by its name. 91 | */ 92 | public void listMetrics(String groupRegex, String nameRegex) { 93 | ToolsUtils.printMetrics(getMetrics(groupRegex, nameRegex)); 94 | } 95 | } 96 | -------------------------------------------------------------------------------- /kukulcan-api/src/test/scala/com/github/mmolimar/kukulcan/KStreamsSpec.scala: -------------------------------------------------------------------------------- 1 | package com.github.mmolimar.kukulcan 2 | 3 | import _root_.com.github.mmolimar.kukulcan.java.{KStreams => JKStreams} 4 | import _root_.com.github.mmolimar.kukulcan.{KStreams => SKStreams} 5 | import org.apache.kafka.common.serialization.Serde 6 | import org.apache.kafka.streams.Topology 7 | import org.apache.kafka.streams.kstream.GlobalKTable 8 | import org.apache.kafka.streams.scala.kstream.{KStream, KTable, Materialized} 9 | import org.apache.kafka.streams.scala.serialization.Serdes 10 | import org.apache.kafka.streams.scala.{ByteArrayKeyValueStore, StreamsBuilder} 11 | 12 | import _root_.java.util.Properties 13 | 14 | class KStreamsSpec extends KukulcanApiTestHarness { 15 | 16 | override def apiClass: Class[_] = classOf[SKStreams] 17 | 18 | override def execScalaTests(): Unit = { 19 | val scalaApi: SKStreams = { 20 | val props = new Properties() 21 | props.put("bootstrap.servers", "localhost:9092") 22 | props.put("client.id", "test-client") 23 | props.put("application.id", "test-app") 24 | props.put("replication.factor", "1") 25 | SKStreams(buildTopology, props) 26 | } 27 | 28 | scalaApi.getMetrics shouldBe scalaApi.getMetrics(".*", ".*") 29 | scalaApi.getMetrics("stream-metrics", "version").size shouldBe 1 30 | 31 | scalaApi.listMetrics() 32 | scalaApi.listMetrics(".*", ".*") 33 | scalaApi.printTopology() 34 | 35 | scalaApi.withApplicationId("test-app2") shouldNot be(None.orNull) 36 | val props = new Properties() 37 | props.put("bootstrap.servers", "localhost:9092") 38 | props.put("application.id", "test-app2") 39 | scalaApi.withProperties(props) 40 | } 41 | 42 | override def execJavaTests(): Unit = { 43 | val javaApi: JKStreams = { 44 | val props = new Properties() 45 | props.put("bootstrap.servers", "localhost:9092") 46 | props.put("client.id", "test-client") 47 | props.put("application.id", "test-app") 48 | props.put("replication.factor", "1") 49 | new JKStreams(buildTopology, props) 50 | } 51 | 52 | javaApi.getMetrics shouldBe javaApi.getMetrics(".*", ".*") 53 | javaApi.getMetrics("stream-metrics", "version").size shouldBe 1 54 | 55 | javaApi.listMetrics() 56 | javaApi.listMetrics(".*", ".*") 57 | javaApi.printTopology() 58 | 59 | javaApi.withApplicationId("test-app2") shouldNot be(None.orNull) 60 | val props = new Properties() 61 | props.put("bootstrap.servers", "localhost:9092") 62 | props.put("application.id", "test-app2") 63 | javaApi.withProperties(props) 64 | } 65 | 66 | private def buildTopology: Topology = { 67 | import org.apache.kafka.streams.scala.ImplicitConversions._ 68 | 69 | implicit def Long: Serde[Long] = Serdes.longSerde 70 | 71 | implicit def String: Serde[String] = Serdes.stringSerde 72 | 73 | val builder = new StreamsBuilder() 74 | val global: GlobalKTable[Long, String] = builder.globalTable[Long, String]( 75 | "sample", 76 | Materialized.as[Long, String, ByteArrayKeyValueStore]("test-global-table") 77 | ) 78 | val records: KStream[Long, String] = builder.stream[Long, String]("test-stream") 79 | 80 | records 81 | .join(global)( 82 | (x, _) => x, 83 | (_, y) => y 84 | ) 85 | .peek((_, _) => {}) 86 | val wordCounts: KTable[String, Long] = records 87 | .flatMapValues(textLine => textLine.toLowerCase.split("\\W+")) 88 | .groupBy((_, word) => word) 89 | .count() 90 | wordCounts.toStream.to("test-stream-result") 91 | 92 | builder.build() 93 | } 94 | 95 | } 96 | -------------------------------------------------------------------------------- /kukulcan-api/src/test/scala/com/github/mmolimar/kukulcan/KAdminAclsSpec.scala: -------------------------------------------------------------------------------- 1 | package com.github.mmolimar.kukulcan 2 | 3 | import _root_.com.github.mmolimar.kukulcan.{KAdmin => SKAdmin} 4 | 5 | import com.github.mmolimar.kukulcan.java.{KAdmin => JKAdmin} 6 | import net.manub.embeddedkafka.{EmbeddedKafka, EmbeddedKafkaConfig} 7 | import org.apache.kafka.clients.admin.KafkaAdminClient 8 | import org.apache.kafka.common.acl._ 9 | import org.apache.kafka.common.resource.{PatternType, ResourcePattern, ResourcePatternFilter, ResourceType} 10 | import org.apache.kafka.common.security.auth.KafkaPrincipal 11 | 12 | import _root_.java.util.Collections._ 13 | import _root_.java.util.Properties 14 | 15 | class KAdminAclsSpec extends KukulcanApiTestHarness with EmbeddedKafka { 16 | 17 | lazy implicit val config: EmbeddedKafkaConfig = EmbeddedKafkaConfig() 18 | 19 | override def apiClass: Class[_] = classOf[SKAdmin] 20 | 21 | override def execScalaTests(): Unit = withRunningKafka { 22 | val scalaApi: SKAdmin = { 23 | val props = new Properties() 24 | props.put("bootstrap.servers", s"localhost:${config.kafkaPort}") 25 | props.put("zookeeper.connect", s"localhost:${config.zooKeeperPort}") 26 | SKAdmin(props) 27 | } 28 | 29 | scalaApi.servers shouldBe s"localhost:${config.kafkaPort}" 30 | scalaApi.client.isInstanceOf[KafkaAdminClient] shouldBe true 31 | 32 | scalaApi.acls.getAcls().isEmpty shouldBe true 33 | 34 | val resourcePattern = new ResourcePattern(ResourceType.TOPIC, "test", PatternType.LITERAL) 35 | val accessControl = 36 | new AccessControlEntry("User:test", "*", AclOperation.DESCRIBE, AclPermissionType.ALLOW) 37 | val acl = new AclBinding(resourcePattern, accessControl) 38 | scalaApi.acls.addAcls(Seq(acl)) 39 | 40 | scalaApi.acls.getAcls().head.pattern().name() shouldBe "test" 41 | scalaApi.acls.listAcls() 42 | val principal = new KafkaPrincipal(KafkaPrincipal.USER_TYPE, "test") 43 | scalaApi.acls.listAcls(principals = Seq(principal)) 44 | 45 | val aclFilter = new AclBindingFilter(ResourcePatternFilter.ANY, AccessControlEntryFilter.ANY) 46 | scalaApi.acls.removeAcls(Seq(aclFilter)) 47 | } 48 | 49 | override def execJavaTests(): Unit = withRunningKafka { 50 | val javaApi: JKAdmin = { 51 | val props = new Properties() 52 | props.put("bootstrap.servers", s"localhost:${config.kafkaPort}") 53 | props.put("zookeeper.connect", s"localhost:${config.zooKeeperPort}") 54 | new JKAdmin(props) 55 | } 56 | 57 | javaApi.servers shouldBe s"localhost:${config.kafkaPort}" 58 | javaApi.client.isInstanceOf[KafkaAdminClient] shouldBe true 59 | 60 | javaApi.acls.getAcls().isEmpty shouldBe true 61 | 62 | val resourcePattern = new ResourcePattern(ResourceType.TOPIC, "test", PatternType.LITERAL) 63 | val accessControl = 64 | new AccessControlEntry("User:test", "*", AclOperation.DESCRIBE, AclPermissionType.ALLOW) 65 | val acl = new AclBinding(resourcePattern, accessControl) 66 | javaApi.acls.addAcls(singletonList(acl)) 67 | javaApi.acls.addAcls(singletonList(acl), true) 68 | 69 | javaApi.acls.getAcls().get(0).pattern().name() shouldBe "test" 70 | javaApi.acls.getAcls(AclBindingFilter.ANY, true).get(0).pattern().name() shouldBe "test" 71 | javaApi.acls.listAcls() 72 | javaApi.acls.listAcls(AclBindingFilter.ANY) 73 | val principal = new KafkaPrincipal(KafkaPrincipal.USER_TYPE, "test") 74 | javaApi.acls.listAcls(singletonList(principal)) 75 | 76 | val aclFilter = new AclBindingFilter(ResourcePatternFilter.ANY, AccessControlEntryFilter.ANY) 77 | javaApi.acls.removeAcls(singletonList(aclFilter)) 78 | javaApi.acls.removeAcls(singletonList(aclFilter), true) 79 | 80 | javaApi.acls.defaultAuthorizer().isPresent shouldBe true 81 | } 82 | 83 | } 84 | -------------------------------------------------------------------------------- /kukulcan-repl/src/main/scala/com/github/mmolimar/kukulcan/package.scala: -------------------------------------------------------------------------------- 1 | package com.github.mmolimar 2 | 3 | import _root_.java.util.Properties 4 | 5 | import com.github.mmolimar.kukulcan.repl._ 6 | import org.apache.kafka.streams.Topology 7 | 8 | package object kukulcan { 9 | 10 | /** 11 | * Create a KAdmin instance reading the {@code admin.properties} file. 12 | * If the instance was already created, it will be reused. 13 | * 14 | * @return The KAdmin instance initialized. 15 | */ 16 | def admin: KAdmin = KAdminApi.inst 17 | 18 | /** 19 | * Create a KConsumer instance reading the {@code consumer.properties} file. 20 | * If the instance was already created, it will be reused. 21 | * 22 | * @return The KConsumer instance initialized. 23 | */ 24 | def consumer[K, V]: KConsumer[K, V] = KConsumerApi.inst.asInstanceOf[KConsumer[K, V]] 25 | 26 | /** 27 | * Create a KProducer instance reading the {@code producer.properties} file. 28 | * If the instance was already created, it will be reused. 29 | * 30 | * @return The KProducer instance initialized. 31 | */ 32 | def producer[K, V]: KProducer[K, V] = KProducerApi.inst.asInstanceOf[KProducer[K, V]] 33 | 34 | /** 35 | * Create a KConnect instance reading the {@code connect.properties} file. 36 | * If the instance was already created, it will be reused. 37 | * 38 | * @return The KConnect instance initialized. 39 | */ 40 | def connect: KConnect = KConnectApi.inst 41 | 42 | /** 43 | * Create a KStreams instance reading the {@code streams.properties} file. 44 | * If the instance was already created, it will be reused. 45 | * 46 | * @param topology The topology to create the KStream 47 | * @return The KStreams instance initialized. 48 | */ 49 | def streams(topology: Topology): KStreams = KStreamsApi.inst(topology) 50 | 51 | /** 52 | * Create a KSchemaRegistry instance reading the {@code schema-registry.properties} file. 53 | * If the instance was already created, it will be reused. 54 | * 55 | * @return The KSchemaRegistry instance initialized. 56 | */ 57 | def schemaRegistry: KSchemaRegistry = KSchemaRegistryApi.inst 58 | 59 | /** 60 | * Create a KKsql instance reading the {@code ksql.properties} file. 61 | * If the instance was already created, it will be reused. 62 | * 63 | * @return The KKsql instance initialized. 64 | */ 65 | def ksql: KKsql = KKsqlApi.inst 66 | 67 | /** 68 | * Re-create all instances using their properties files. 69 | * 70 | */ 71 | def reload(): Unit = { 72 | KAdminApi.reload() 73 | KConsumerApi.reload() 74 | KProducerApi.reload() 75 | KConnectApi.reload() 76 | KStreamsApi.reload() 77 | KSchemaRegistryApi.reload() 78 | KKsqlApi.reload() 79 | println("Done!") 80 | } 81 | 82 | private[kukulcan] object KAdminApi extends KApi[KAdmin]("admin") { 83 | override protected def createInstance(props: Properties): KAdmin = KAdmin(props) 84 | } 85 | 86 | private[kukulcan] object KConsumerApi extends KApi[KConsumer[AnyRef, AnyRef]]("consumer") { 87 | override protected def createInstance(props: Properties): KConsumer[AnyRef, AnyRef] = KConsumer(props) 88 | } 89 | 90 | private[kukulcan] object KProducerApi extends KApi[KProducer[AnyRef, AnyRef]]("producer") { 91 | override protected def createInstance(props: Properties): KProducer[AnyRef, AnyRef] = KProducer(props) 92 | } 93 | 94 | private[kukulcan] object KConnectApi extends KApi[KConnect]("connect") { 95 | override protected def createInstance(props: Properties): KConnect = KConnect(props) 96 | } 97 | 98 | private[kukulcan] object KStreamsApi extends KApi[Topology => KStreams]("streams") { 99 | override protected def createInstance(props: Properties): Topology => KStreams = { 100 | topology: Topology => KStreams(topology, props) 101 | } 102 | } 103 | 104 | private[kukulcan] object KSchemaRegistryApi extends KApi[KSchemaRegistry]("schema-registry") { 105 | override protected def createInstance(props: Properties): KSchemaRegistry = KSchemaRegistry(props) 106 | } 107 | 108 | private[kukulcan] object KKsqlApi extends KApi[KKsql]("ksql") { 109 | override protected def createInstance(props: Properties): KKsql = KKsql(props) 110 | } 111 | 112 | } 113 | -------------------------------------------------------------------------------- /kukulcan-repl/src/test/scala/com/github/mmolimar/kukulcan/repl/KukulcanReplSpec.scala: -------------------------------------------------------------------------------- 1 | package com.github.mmolimar.kukulcan.repl 2 | 3 | import org.apache.kafka.common.serialization.Serde 4 | import org.apache.kafka.streams.scala.StreamsBuilder 5 | import org.apache.kafka.streams.scala.serialization.Serdes 6 | import org.scalatest.matchers.should.Matchers 7 | import org.scalatest.wordspec.AnyWordSpecLike 8 | 9 | class KukulcanReplSpec extends AnyWordSpecLike with Matchers { 10 | 11 | s"a Kukulcan Scala REPL" when { 12 | 13 | "initializing" should { 14 | val interpreter = new KukulcanILoop() 15 | interpreter.settings = KukulcanRepl.buildSettings 16 | interpreter.createInterpreter() 17 | 18 | "have an interpreter" in { 19 | interpreter.intp shouldNot be(None.orNull) 20 | interpreter.printWelcome() 21 | interpreter.prompt shouldBe "@ " 22 | } 23 | } 24 | 25 | "using the KApi" should { 26 | 27 | "reload services" in { 28 | import com.github.mmolimar.kukulcan 29 | import org.apache.kafka.streams.scala.ImplicitConversions._ 30 | 31 | implicit def String: Serde[String] = Serdes.stringSerde 32 | 33 | val admin = kukulcan.admin 34 | val connect = kukulcan.connect 35 | val consumer = kukulcan.consumer 36 | val producer = kukulcan.producer 37 | val topology = { 38 | val streamsBuilder = new StreamsBuilder() 39 | streamsBuilder.stream[String, String]("test") 40 | streamsBuilder.build() 41 | } 42 | val streams = { 43 | kukulcan.streams(topology) 44 | } 45 | val ksql = kukulcan.ksql 46 | val schemaRegistry = kukulcan.schemaRegistry 47 | 48 | admin shouldBe kukulcan.admin 49 | connect shouldBe kukulcan.connect 50 | consumer shouldBe kukulcan.consumer 51 | producer shouldBe kukulcan.producer 52 | streams shouldNot be(kukulcan.streams(topology)) 53 | ksql shouldBe kukulcan.ksql 54 | schemaRegistry shouldBe kukulcan.schemaRegistry 55 | 56 | kukulcan.reload() 57 | 58 | admin shouldNot be(kukulcan.admin) 59 | connect shouldNot be(kukulcan.connect) 60 | consumer shouldNot be(kukulcan.consumer) 61 | producer shouldNot be(kukulcan.producer) 62 | streams shouldNot be(kukulcan.streams(topology)) 63 | ksql shouldNot be(kukulcan.ksql) 64 | schemaRegistry shouldNot be(kukulcan.schemaRegistry) 65 | } 66 | } 67 | } 68 | 69 | s"a Kukulcan Java REPL" when { 70 | "initializing" should { 71 | 72 | "have default args" in { 73 | val args = JKukulcanRepl.shellArgs(Array.empty) 74 | args.contains("--feedback") shouldBe true 75 | args.contains("concise") shouldBe true 76 | } 77 | } 78 | 79 | "using the KApi" should { 80 | 81 | "reload services" in { 82 | import com.github.mmolimar.kukulcan.java._ 83 | import org.apache.kafka.streams.scala.ImplicitConversions._ 84 | 85 | implicit def String: Serde[String] = Serdes.stringSerde 86 | 87 | val admin = Kukulcan.admin 88 | val connect = Kukulcan.connect 89 | val consumer = Kukulcan.consumer 90 | val producer = Kukulcan.producer 91 | val topology = { 92 | val streamsBuilder = new StreamsBuilder() 93 | streamsBuilder.stream[String, String]("test") 94 | streamsBuilder.build() 95 | } 96 | val streams = { 97 | Kukulcan.streams(topology) 98 | } 99 | val ksql = Kukulcan.ksql 100 | val schemaRegistry = Kukulcan.schemaRegistry 101 | 102 | admin shouldBe Kukulcan.admin 103 | connect shouldBe Kukulcan.connect 104 | consumer shouldBe Kukulcan.consumer 105 | producer shouldBe Kukulcan.producer 106 | streams shouldNot be(Kukulcan.streams(topology)) 107 | ksql shouldBe Kukulcan.ksql 108 | schemaRegistry shouldBe Kukulcan.schemaRegistry 109 | 110 | Kukulcan.reload() 111 | 112 | admin shouldNot be(Kukulcan.admin) 113 | connect shouldNot be(Kukulcan.connect) 114 | consumer shouldNot be(Kukulcan.consumer) 115 | producer shouldNot be(Kukulcan.producer) 116 | streams shouldNot be(Kukulcan.streams(topology)) 117 | ksql shouldNot be(Kukulcan.ksql) 118 | schemaRegistry shouldNot be(Kukulcan.schemaRegistry) 119 | } 120 | } 121 | } 122 | } 123 | -------------------------------------------------------------------------------- /kukulcan-api/src/main/java/com/github/mmolimar/kukulcan/java/KStreams.java: -------------------------------------------------------------------------------- 1 | package com.github.mmolimar.kukulcan.java; 2 | 3 | import org.apache.kafka.common.Metric; 4 | import org.apache.kafka.common.MetricName; 5 | import org.apache.kafka.streams.KafkaStreams; 6 | import org.apache.kafka.streams.Topology; 7 | 8 | import java.util.Map; 9 | import java.util.Properties; 10 | 11 | import static com.github.mmolimar.kukulcan.java.KUtils.toJavaMap; 12 | 13 | /** 14 | * An enriched implementation of the {@code org.apache.kafka.streams.KafkaStreams} class to 15 | * manages streams in Kafka. 16 | */ 17 | public class KStreams extends KafkaStreams { 18 | 19 | private final com.github.mmolimar.kukulcan.KStreams kstreams; 20 | 21 | /** 22 | * @param topology Topology specifying the computational logic. 23 | * @param props Properties with the configuration. 24 | */ 25 | public KStreams(Topology topology, Properties props) { 26 | super(topology, props); 27 | this.kstreams = new com.github.mmolimar.kukulcan.KStreams(topology, props); 28 | } 29 | 30 | private KStreams(com.github.mmolimar.kukulcan.KStreams kstreams) { 31 | super(kstreams.topology(), kstreams.props()); 32 | this.kstreams = kstreams; 33 | } 34 | 35 | /** 36 | * Create new instance with a new application id 37 | * 38 | * @param applicationId The new application id. 39 | * @return The new instance created. 40 | */ 41 | public KStreams withApplicationId(String applicationId) { 42 | return new KStreams(this.kstreams.withApplicationId(applicationId)); 43 | } 44 | 45 | /** 46 | * Create new instance with the properties specified. 47 | * 48 | * @param props The properties to create the instance. 49 | * @return The new instance created. 50 | */ 51 | public KStreams withProperties(Properties props) { 52 | return new KStreams(this.kstreams.withProperties(props)); 53 | } 54 | 55 | /** 56 | * Get all metrics registered. 57 | * 58 | * @return a {@code Map[MetricName, Metric]} with the all metrics registered. 59 | */ 60 | public Map getMetrics() { 61 | return getMetrics(".*"); 62 | } 63 | 64 | /** 65 | * Get all metrics registered filtered by the group regular expressions. 66 | * 67 | * @param groupRegex Regex to filter metrics by group name. 68 | * @return a {@code Map[MetricName, Metric]} with the all metrics registered filtered by the group regular 69 | * expression. 70 | */ 71 | public Map getMetrics(String groupRegex) { 72 | return getMetrics(groupRegex, ".*"); 73 | } 74 | 75 | /** 76 | * Get all metrics registered filtered by the group and name regular expressions. 77 | * 78 | * @param groupRegex Regex to filter metrics by group name. 79 | * @param nameRegex Regex to filter metrics by its name. 80 | * @return a {@code Map[MetricName, Metric]} with the all metrics registered filtered by the group and name regular 81 | * expressions. 82 | */ 83 | public Map getMetrics(String groupRegex, String nameRegex) { 84 | return toJavaMap(this.kstreams.getMetrics(groupRegex, nameRegex)); 85 | } 86 | 87 | /** 88 | * Print all metrics. 89 | */ 90 | public void listMetrics() { 91 | listMetrics(".*"); 92 | } 93 | 94 | /** 95 | * Print all metrics filtered by the group regular expression. 96 | * 97 | * @param groupRegex Regex to filter metrics by group name. 98 | */ 99 | public void listMetrics(String groupRegex) { 100 | listMetrics(groupRegex, ".*"); 101 | } 102 | 103 | /** 104 | * Print all metrics filtered by the group and name regular expressions. 105 | * 106 | * @param groupRegex Regex to filter metrics by group name. 107 | * @param nameRegex Regex to filter metrics by its name. 108 | */ 109 | public void listMetrics(String groupRegex, String nameRegex) { 110 | this.kstreams.listMetrics(groupRegex, nameRegex); 111 | } 112 | 113 | /** 114 | * Print the topology in an ASCII graph. 115 | */ 116 | public void printTopology() { 117 | printTopology(true, true); 118 | } 119 | 120 | /** 121 | * Print the topology in an ASCII graph. 122 | * 123 | * @param subtopologies If to include the subtopologies. 124 | * @param globalStores If to include the global stores. 125 | */ 126 | public void printTopology(boolean subtopologies, boolean globalStores) { 127 | this.kstreams.printTopology(subtopologies, globalStores); 128 | } 129 | 130 | } 131 | -------------------------------------------------------------------------------- /kukulcan-repl/src/main/java/com/github/mmolimar/kukulcan/java/Kukulcan.java: -------------------------------------------------------------------------------- 1 | package com.github.mmolimar.kukulcan.java; 2 | 3 | import com.github.mmolimar.kukulcan.KKsql; 4 | import com.github.mmolimar.kukulcan.repl.KApi; 5 | import org.apache.kafka.streams.Topology; 6 | 7 | import java.util.Properties; 8 | import java.util.function.Function; 9 | 10 | public class Kukulcan { 11 | 12 | private static KApi kadminApi = new KApi<>("admin") { 13 | @Override 14 | public KAdmin createInstance(Properties props) { 15 | return new KAdmin(props); 16 | } 17 | }; 18 | 19 | private static KApi> kconsumerApi = new KApi<>("consumer") { 20 | @Override 21 | public KConsumer createInstance(Properties props) { 22 | return new KConsumer<>(props); 23 | } 24 | }; 25 | 26 | private static KApi> kproducerApi = new KApi<>("producer") { 27 | @Override 28 | public KProducer createInstance(Properties props) { 29 | return new KProducer<>(props); 30 | } 31 | }; 32 | 33 | private static KApi kconnectApi = new KApi<>("connect") { 34 | @Override 35 | public KConnect createInstance(Properties props) { 36 | return new KConnect(props); 37 | } 38 | }; 39 | 40 | private static KApi> kstreamsApi = new KApi<>("streams") { 41 | @Override 42 | public Function createInstance(Properties props) { 43 | return (topology -> new KStreams(topology, props)); 44 | } 45 | }; 46 | 47 | private static KApi kschemaRegistryApi = new KApi<>("schema-registry") { 48 | @Override 49 | public KSchemaRegistry createInstance(Properties props) { 50 | return new KSchemaRegistry(props); 51 | } 52 | }; 53 | 54 | private static KApi kksqlApi = new KApi<>("ksql") { 55 | @Override 56 | public KKsql createInstance(Properties props) { 57 | return new KKsql(props); 58 | } 59 | }; 60 | 61 | /** 62 | * Create a KAdmin instance reading the {@code admin.properties} file. 63 | * If the instance was already created, it will be reused. 64 | * 65 | * @return The KAdmin instance initialized. 66 | */ 67 | public static KAdmin admin() { 68 | return kadminApi.inst(); 69 | } 70 | 71 | /** 72 | * Create a KConsumer instance reading the {@code consumer.properties} file. 73 | * If the instance was already created, it will be reused. 74 | * 75 | * @return The KConsumer instance initialized. 76 | */ 77 | @SuppressWarnings("unchecked") 78 | public static KConsumer consumer() { 79 | return (KConsumer) kconsumerApi.inst(); 80 | } 81 | 82 | /** 83 | * Create a KProducer instance reading the {@code producer.properties} file. 84 | * If the instance was already created, it will be reused. 85 | * 86 | * @return The KProducer instance initialized. 87 | */ 88 | @SuppressWarnings("unchecked") 89 | public static KProducer producer() { 90 | return (KProducer) kproducerApi.inst(); 91 | } 92 | 93 | /** 94 | * Create a KConnect instance reading the {@code connect.properties} file. 95 | * If the instance was already created, it will be reused. 96 | * 97 | * @return The KConnect instance initialized. 98 | */ 99 | public static KConnect connect() { 100 | return kconnectApi.inst(); 101 | } 102 | 103 | /** 104 | * Create a KStreams instance reading the {@code streams.properties} file. 105 | * If the instance was already created, it will be reused. 106 | * 107 | * @param topology The topology to create the KStream 108 | * @return The KStreams instance initialized. 109 | */ 110 | public static KStreams streams(Topology topology) { 111 | return kstreamsApi.inst().apply(topology); 112 | } 113 | 114 | /** 115 | * Create a KSchemaRegistry instance reading the {@code schema-registry.properties} file. 116 | * If the instance was already created, it will be reused. 117 | * 118 | * @return The KSchemaRegistry instance initialized. 119 | */ 120 | public static KSchemaRegistry schemaRegistry() { 121 | return kschemaRegistryApi.inst(); 122 | } 123 | 124 | /** 125 | * Create a KKsql instance reading the {@code ksql.properties} file. 126 | * If the instance was already created, it will be reused. 127 | * 128 | * @return The KKsql instance initialized. 129 | */ 130 | public static KKsql ksql() { 131 | return kksqlApi.inst(); 132 | } 133 | 134 | /** 135 | * Re-create all instances using their properties files. 136 | * 137 | */ 138 | public static void reload() { 139 | kadminApi.reload(); 140 | kconsumerApi.reload(); 141 | kproducerApi.reload(); 142 | kconnectApi.reload(); 143 | kstreamsApi.reload(); 144 | kschemaRegistryApi.reload(); 145 | kksqlApi.reload(); 146 | System.out.println("Done!"); 147 | } 148 | } 149 | -------------------------------------------------------------------------------- /kukulcan-api/src/test/scala/com/github/mmolimar/kukulcan/KKsqlSpec.scala: -------------------------------------------------------------------------------- 1 | package com.github.mmolimar.kukulcan 2 | 3 | import _root_.com.github.mmolimar.kukulcan.java.{KKsql => JKKsql} 4 | import _root_.com.github.mmolimar.kukulcan.{KKsql => SKKsql} 5 | import io.confluent.ksql.rest.entity.{CommandStatus, CommandStatusEntity} 6 | import io.confluent.ksql.rest.server.{KsqlRestApplication, KsqlRestConfig} 7 | import io.confluent.ksql.util.KsqlException 8 | import net.manub.embeddedkafka.{EmbeddedKafka, EmbeddedKafkaConfig} 9 | 10 | import _root_.java.util.Properties 11 | import _root_.java.util.Collections 12 | 13 | class KKsqlSpec extends KukulcanApiTestHarness with EmbeddedKafka { 14 | 15 | lazy implicit val config: EmbeddedKafkaConfig = EmbeddedKafkaConfig() 16 | 17 | val ksqlPort = 37073 18 | val ksqlListener = s"http://localhost:$ksqlPort" 19 | 20 | override def apiClass: Class[_] = classOf[SKKsql] 21 | 22 | override def execScalaTests(): Unit = withRunningKafka { 23 | withKsqlServer { 24 | val scalaApi: SKKsql = { 25 | val props = new Properties() 26 | props.put("ksql.server", ksqlListener) 27 | props.put("ksql.credentials.user", "testuser") 28 | props.put("ksql.credentials.password", "testpassword") 29 | SKKsql(props) 30 | } 31 | scalaApi.getServerAddress.toString shouldBe ksqlListener 32 | scalaApi.getServerInfo.getServerStatus shouldBe "RUNNING" 33 | scalaApi.getServerMetadata.getVersion shouldBe "6.1.0" 34 | scalaApi.getServerMetadataId.getId shouldBe "" 35 | scalaApi.getServerHealth.getIsHealthy shouldBe true 36 | scalaApi.getAllStatuses.size() shouldBe 0 37 | 38 | scalaApi.setProperty("ksql.streams.auto.offset.reset", "earliest") shouldBe None.orNull 39 | scalaApi.unsetProperty("ksql.streams.auto.offset.reset").toString shouldBe "earliest" 40 | 41 | val ksqlCreate = "CREATE TABLE TEST (ID VARCHAR PRIMARY KEY, FIELD1 VARCHAR) WITH (KAFKA_TOPIC = 'test', VALUE_FORMAT = 'DELIMITED');" 42 | val createResult = scalaApi.makeKsqlRequest(ksqlCreate) 43 | scalaApi.getStatus(createResult.get(0).asInstanceOf[CommandStatusEntity].getCommandId.toString).getStatus shouldBe 44 | CommandStatus.Status.SUCCESS 45 | 46 | val ksqlInsert = "INSERT INTO TEST (ID, FIELD1) VALUES ('test', 'value');" 47 | scalaApi.makeKsqlRequest(ksqlInsert).size() shouldBe 0 48 | scalaApi.getAllStatuses.size() shouldBe 1 49 | 50 | val ksqlQuery = "SELECT * FROM TEST EMIT CHANGES LIMIT 1;" 51 | scalaApi.makeQueryRequest(ksqlQuery).size shouldBe 3 52 | scalaApi.makeQueryRequestStreamed(ksqlQuery) shouldNot be(None.orNull) 53 | 54 | val ksqlPrint = "PRINT test FROM BEGINNING INTERVAL 1 LIMIT 1;" 55 | scalaApi.makePrintTopicRequestStreamed(ksqlPrint) shouldNot be(None.orNull) 56 | scalaApi.makePrintTopicRequest(ksqlPrint).size shouldBe 3 57 | 58 | assertThrows[KsqlException] { 59 | scalaApi.makeHeartbeatRequest 60 | } 61 | assertThrows[KsqlException] { 62 | scalaApi.makeClusterStatusRequest 63 | } 64 | } 65 | } 66 | 67 | override def execJavaTests(): Unit = withRunningKafka { 68 | withKsqlServer { 69 | val javaApi: JKKsql = { 70 | val props = new Properties() 71 | props.put("ksql.server", ksqlListener) 72 | new JKKsql(props) 73 | } 74 | 75 | javaApi.getServerAddress.toString shouldBe ksqlListener 76 | javaApi.getServerInfo.getServerStatus shouldBe "RUNNING" 77 | javaApi.getServerMetadata.getVersion shouldBe "6.1.0" 78 | javaApi.getServerMetadataId.getId shouldBe "" 79 | javaApi.getServerHealth.getIsHealthy shouldBe true 80 | javaApi.getAllStatuses.size() shouldBe 0 81 | 82 | javaApi.setProperty("ksql.streams.auto.offset.reset", "earliest") shouldBe None.orNull 83 | javaApi.unsetProperty("ksql.streams.auto.offset.reset").toString shouldBe "earliest" 84 | 85 | val ksqlCreate = "CREATE TABLE TEST (ID VARCHAR PRIMARY KEY, FIELD1 VARCHAR) WITH (KAFKA_TOPIC = 'test', VALUE_FORMAT = 'DELIMITED');" 86 | val createResult = javaApi.makeKsqlRequest(ksqlCreate, -1L) 87 | javaApi.getStatus(createResult.get(0).asInstanceOf[CommandStatusEntity].getCommandId.toString).getStatus shouldBe 88 | CommandStatus.Status.SUCCESS 89 | 90 | val ksqlInsert = "INSERT INTO TEST (ID, FIELD1) VALUES ('test', 'value');" 91 | javaApi.makeKsqlRequest(ksqlInsert).size() shouldBe 0 92 | javaApi.getAllStatuses.size() shouldBe 1 93 | 94 | val ksqlQuery = "SELECT * FROM TEST EMIT CHANGES LIMIT 1;" 95 | javaApi.makeQueryRequest(ksqlQuery).size shouldBe 3 96 | javaApi.makeQueryRequest(ksqlQuery, Collections.emptyMap(), Collections.emptyMap()).size shouldBe 3 97 | javaApi.makeQueryRequestStreamed(ksqlQuery) shouldNot be(None.orNull) 98 | 99 | val ksqlPrint = "PRINT test FROM BEGINNING INTERVAL 1 LIMIT 1;" 100 | javaApi.makePrintTopicRequestStreamed(ksqlPrint) shouldNot be(None.orNull) 101 | javaApi.makePrintTopicRequest(ksqlPrint).size shouldBe 3 102 | 103 | assertThrows[KsqlException] { 104 | javaApi.makeHeartbeatRequest 105 | } 106 | assertThrows[KsqlException] { 107 | javaApi.makeClusterStatusRequest 108 | } 109 | } 110 | } 111 | 112 | private def withKsqlServer(body: => Unit): Unit = { 113 | val ksqlServer = { 114 | val props = new Properties() 115 | props.put("listeners", ksqlListener) 116 | props.put("bootstrap.servers", s"localhost:${config.kafkaPort}") 117 | val ksqlRestConfig = new KsqlRestConfig(props) 118 | KsqlRestApplication.buildApplication(ksqlRestConfig) 119 | } 120 | ksqlServer.startAsync() 121 | 122 | val kadmin = { 123 | val props = new Properties() 124 | props.put("bootstrap.servers", s"localhost:${config.kafkaPort}") 125 | KAdmin(props) 126 | } 127 | kadmin.topics.createTopic("test", 1, 1) 128 | 129 | try { 130 | body 131 | } finally { 132 | ksqlServer.notifyTerminated() 133 | ksqlServer.shutdown() 134 | } 135 | } 136 | 137 | } 138 | -------------------------------------------------------------------------------- /kukulcan-api/src/test/scala/com/github/mmolimar/kukulcan/KAdminTopicsSpec.scala: -------------------------------------------------------------------------------- 1 | package com.github.mmolimar.kukulcan 2 | 3 | import _root_.com.github.mmolimar.kukulcan.{KAdmin => SKAdmin} 4 | 5 | import com.github.mmolimar.kukulcan.java.{KUtils, KAdmin => JKAdmin} 6 | import net.manub.embeddedkafka.{EmbeddedKafka, EmbeddedKafkaConfig} 7 | import org.apache.kafka.clients.admin.{CreateTopicsOptions, KafkaAdminClient, NewTopic} 8 | 9 | import _root_.java.time.Duration 10 | import _root_.java.util.{Properties, Arrays => JArrays, HashSet => JSet} 11 | 12 | class KAdminTopicsSpec extends KukulcanApiTestHarness with EmbeddedKafka { 13 | 14 | lazy implicit val config: EmbeddedKafkaConfig = EmbeddedKafkaConfig() 15 | 16 | override def apiClass: Class[_] = classOf[SKAdmin] 17 | 18 | override def execScalaTests(): Unit = withRunningKafka { 19 | val scalaApi: SKAdmin = { 20 | val props = new Properties() 21 | props.put("bootstrap.servers", s"localhost:${config.kafkaPort}") 22 | SKAdmin(props) 23 | } 24 | val kconsumer = { 25 | val props = new Properties() 26 | props.put("bootstrap.servers", s"localhost:${config.kafkaPort}") 27 | props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer") 28 | props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer") 29 | props.put("group.id", "test-group") 30 | KConsumer[String, String](props) 31 | } 32 | 33 | scalaApi.servers shouldBe s"localhost:${config.kafkaPort}" 34 | scalaApi.client.isInstanceOf[KafkaAdminClient] shouldBe true 35 | 36 | scalaApi.topics.getTopics(excludeInternalTopics = true).size shouldBe 0 37 | scalaApi.topics.listTopics() 38 | 39 | scalaApi.topics.createTopic("topic1", 1, 1) shouldBe true 40 | scalaApi.topics.createTopicWithReplicasAssignments("topic2", Map(0 -> Seq(0))) shouldBe true 41 | scalaApi.topics.createTopics(Seq(new NewTopic("topic1", 1, 1.toShort))) shouldBe true 42 | scalaApi.topics.createTopics(Seq(new NewTopic("topic3", 1, 1.toShort))) shouldBe true 43 | 44 | kconsumer.subscribe("topic1") 45 | kconsumer.poll(Duration.ofMillis(2000)) 46 | 47 | scalaApi.topics.getTopics(Some("topic1")) shouldBe Seq("topic1") 48 | scalaApi.topics.getTopics(None).toSet shouldBe 49 | Seq("topic1", "topic2", "topic3", "__consumer_offsets").toSet 50 | scalaApi.topics.getTopics(None, excludeInternalTopics = true) shouldBe Seq("topic1", "topic2", "topic3") 51 | scalaApi.topics.listTopics() 52 | 53 | scalaApi.topics.getTopicAndPartitionDescription("topic1", None).get._1.topic shouldBe "topic1" 54 | scalaApi.topics.getTopicAndPartitionDescription(Seq("topic1")).head._1.topic shouldBe "topic1" 55 | 56 | scalaApi.topics.describeTopic("topic1") 57 | scalaApi.topics.describeTopics(Seq("topic1")) 58 | 59 | scalaApi.topics.getNewPartitionsDistribution("topic1", 1, Map(0 -> Seq(0))) 60 | .head._2.totalCount() shouldBe 1 61 | scalaApi.topics.getNewPartitionsDistribution(Seq("topic1"), 2, Map(0 -> Seq(0))) 62 | .head._2.totalCount() shouldBe 2 63 | 64 | scalaApi.topics.alterTopic("topic1", 2, Map(0 -> Seq(0), 1 -> Seq(0))) 65 | scalaApi.topics.alterTopics(Seq("topic2"), 2, Map(0 -> Seq(0), 1 -> Seq(0))) 66 | 67 | scalaApi.topics.deleteTopic("topic1") 68 | scalaApi.topics.deleteTopic("non-existent") 69 | scalaApi.topics.deleteTopics(Seq("topic2", "topic3")) 70 | scalaApi.topics.getTopics(None, excludeInternalTopics = true).size shouldBe 0 71 | } 72 | 73 | override def execJavaTests(): Unit = withRunningKafka { 74 | val javaApi: JKAdmin = { 75 | val props = new Properties() 76 | props.put("bootstrap.servers", s"localhost:${config.kafkaPort}") 77 | new JKAdmin(props) 78 | } 79 | val kconsumer = { 80 | val props = new Properties() 81 | props.put("bootstrap.servers", s"localhost:${config.kafkaPort}") 82 | props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer") 83 | props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer") 84 | props.put("group.id", "test-group") 85 | KConsumer[String, String](props) 86 | } 87 | 88 | javaApi.servers shouldBe s"localhost:${config.kafkaPort}" 89 | javaApi.client.isInstanceOf[KafkaAdminClient] shouldBe true 90 | 91 | javaApi.topics().getTopics(true).size shouldBe 0 92 | javaApi.topics().listTopics() 93 | 94 | javaApi.topics().createTopic("topic1", 1, 1) shouldBe true 95 | val replicaAssignments = KUtils.toJavaMap(Map(Integer.valueOf(0) -> JArrays.asList(Integer.valueOf(0)))) 96 | javaApi.topics() createTopicWithReplicasAssignments("topic2", replicaAssignments) shouldBe true 97 | javaApi.topics().createTopics( 98 | JArrays.asList(new NewTopic("topic1", 1, 1.toShort))) shouldBe true 99 | javaApi.topics().createTopics( 100 | JArrays.asList(new NewTopic("topic3", 1, 1.toShort)), new CreateTopicsOptions) shouldBe true 101 | 102 | kconsumer.subscribe("topic1") 103 | kconsumer.poll(Duration.ofMillis(2000)) 104 | 105 | javaApi.topics().getTopics("topic1", false) shouldBe JArrays.asList("topic1") 106 | new JSet[String](javaApi.topics().getTopics(false)) shouldBe 107 | new JSet[String](JArrays.asList("topic1", "topic2", "topic3", "__consumer_offsets")) 108 | new JSet[String](javaApi.topics().getTopics(true)) shouldBe 109 | new JSet[String](JArrays.asList("topic1", "topic2", "topic3")) 110 | javaApi.topics().listTopics(false) 111 | 112 | javaApi.topics().describeTopic("topic1") 113 | javaApi.topics().describeTopics(JArrays.asList("topic1")) 114 | 115 | var replicaAssignment = KUtils.toJavaMap(Map(Integer.valueOf(0) -> JArrays.asList(Integer.valueOf(0)))) 116 | javaApi.topics().getNewPartitionsDistribution("topic1", 1, replicaAssignment) 117 | .get("topic1").totalCount() shouldBe 1 118 | javaApi.topics().getNewPartitionsDistribution(JArrays.asList("topic1"), 2, replicaAssignment) 119 | .get("topic1").totalCount() shouldBe 2 120 | 121 | replicaAssignment = KUtils.toJavaMap(Map( 122 | Integer.valueOf(0) -> JArrays.asList(Integer.valueOf(0)), 123 | Integer.valueOf(1) -> JArrays.asList(Integer.valueOf(0)), 124 | )) 125 | javaApi.topics().alterTopic("topic1", 2, replicaAssignment) 126 | javaApi.topics().alterTopics(JArrays.asList("topic2"), 2, replicaAssignment) 127 | 128 | javaApi.topics().deleteTopic("topic1") 129 | javaApi.topics().deleteTopic("non-existent") 130 | javaApi.topics().deleteTopics(JArrays.asList("topic2", "topic3")) 131 | javaApi.topics().getTopics(true).size shouldBe 0 132 | } 133 | 134 | } 135 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # kukulcan [![Build Status](https://circleci.com/gh/mmolimar/kukulcan.svg?style=shield)](https://circleci.com/gh/mmolimar/kukulcan) 2 | 3 | *K'uk'ulkan* ("Feathered Serpent") is the name of a deity which was worshipped by the Yucatec maya people. You can 4 | read a lot more in books or on the Internet about it and will see that, in someways, is related to the wind and water. 5 | 6 | Beyond the origin of this name I reused to name this project, Kukulcan provides an API and different 7 | sort of [REPLs](https://en.wikipedia.org/wiki/Read%E2%80%93eval%E2%80%93print_loop) to interact with your streams 8 | or administer your [Apache Kafka](https://kafka.apache.org) deployment. 9 | 10 | It supports POSIX and Windows OS and Scala, Java and Python programming languages. 11 | 12 | ![](/docs/img/kukulcan.png) 13 | 14 | ## Motivation 15 | 16 | As Kafka users (developers, devops, etc.), we have to interact with it in different ways. Maybe using the 17 | command-line tools it provides, an external tool to look for a specific info, an IDE to develop a snippet 18 | in order to test something... but what if you could do all those things in the same interface with the 19 | flexibility of a REPL to do exactly what you need? 20 | 21 | This is the aim of this project: enabling a simple interface to interact with your Kafka cluster taking advantage 22 | of its rich API plus some additional utils included and make your interaction better. 23 | 24 | ## Getting started 25 | 26 | ### Requirements 27 | 28 | Before starting, you'll need to have installed JDK 11 and [SBT](https://www.scala-sbt.org/). Additionally, 29 | if you want to use the PyKukulcan REPL, you'll also need Python (tested 2.7, 3.4, 3.5, 3.6, and 3.7 versions) 30 | and [pip](https://pypi.org/project/pip) installed. 31 | 32 | ### Building from source 33 | 34 | Just clone the ``kukulcan`` repo and *"kukulcan-it"*: 35 | 36 | ``git clone https://github.com/mmolimar/kukulcan.git && cd kukulcan`` 37 | 38 | ``sbt kukulcan`` 39 | 40 | ### Configuration 41 | 42 | In the ``config`` directory, you'll find some files with the configurations for each of the APIs Kukulcan provides. 43 | All the possible configs for each API are in these files with their description so, if you need to set some 44 | specific configs for your environment, you should set them there before starting. 45 | 46 | Also, there is an important environment variable named **``KUKULCAN_HOME``**. If not set, its default value will be 47 | the root folder of the cloned project. When starting a Kukulcan REPL, it will look for this environment variable 48 | and read the properties files contained in its ``config`` subdirectory. 49 | 50 | ## Modules 51 | 52 | The project contains three subprojects or modules described below but, if you generate the Scala docs executing 53 | ``sbt doc`` in the command line, you'll be able to see a more detailed description. 54 | 55 | ### kukulcan-api 56 | 57 | Contains all the Scala and Java classes to interact with Kafka and extending the Kafka API to enable new 58 | functionalities. This API contains: 59 | 60 | * **KAdmin**: grouped utils for administrative operations for topics, configs, ACLs and metrics. 61 | * **KConnect**: methods to execute requests against Kafka Connect REST API. 62 | * **KConsumer** and **KProducer**: Kafka consumer/producer with some extra features. 63 | * **KStreams**: extends Kafka Streams to see how your topology is (printing it in a graph). 64 | * **KSchemaRegistry**: to interact and manage schemas in [Confluent Schema Registry](https://github.com/confluentinc/schema-registry). 65 | * **KKsql**: client for querying [Confluent KSQL](https://github.com/confluentinc/ksql) server and integrated with the KSQL-CLI. 66 | 67 | ### kukulcan-repl 68 | 69 | Enables two sort of entry points for the REPLs: one based on the [Scala REPL](https://docs.scala-lang.org/overviews/repl/overview.html) 70 | and the other based on [JShell](https://docs.oracle.com/javase/9/jshell). 71 | 72 | Additionally, it includes the logic to read and ``reload`` your configurations stored the ``config`` directory 73 | transparently. 74 | 75 | ### pykukulcan 76 | 77 | This module includes the needed bindings to use Kukulcan with Python via [py4j](https://www.py4j.org). 78 | By now, the only bindings available are for the REPL (in Java). 79 | 80 | ## Running the REPL 81 | 82 | Kukulcan takes advantage of multiple functionalities we already have in Java, Scala and other tools to build a 83 | custom REPL. Actually, the project enables four type of REPLs with Ammonite, Scala, JShell and Python. 84 | 85 | After building the source, you'll be able to start any of the following REPL options. Here are a few examples 86 | you can do with Kukulcan: 87 | 88 | - Graphical representation of a topology in Kafka Streams: 89 | 90 | ![](/docs/img/kstreams.png) 91 | 92 | - Kafka Connect interaction: 93 | 94 | ![](/docs/img/kconnect.png) 95 | 96 | - Managing schemas in Schema Registry: 97 | 98 | ![](/docs/img/kschema-registry.png) 99 | 100 | - Interacting with KSQL server and using the KSQL-CLI: 101 | 102 | ![](/docs/img/kksql.png) 103 | 104 | > **NOTE**: The REPLs have already the Kukulcan imports: ``com.github.mmolimar.kukulcan`` in case of the 105 | Scala and Ammonite REPLs and ``com.github.mmolimar.kukulcan.Kukulcan`` in case of the JShell REPL. So you just 106 | have to start typing ``kukulcan.