├── .hgignore ├── MANIFEST.in ├── README.md ├── django_pgjsonb ├── __init__.py └── fields.py ├── setup.cfg └── setup.py /.hgignore: -------------------------------------------------------------------------------- 1 | syntax:glob 2 | *.egg-info 3 | dist 4 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | include README.md 2 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # django-pgjsonb 2 | Django Postgres JSONB Fields support with lookups 3 | 4 | Originaly inspired by [django-postgres](https://bitbucket.org/schinckel/django-postgres/) 5 | 6 | 7 | Change Logs 8 | =========== 9 | 2017-09-13: 0.0.29 10 | Fix JsonAdapter Python2 incompatible 11 | 12 | 2017-09-11: 0.0.28 13 | Fix contained_by contains empty {} 14 | Fix error of has_any/ has_all 15 | Fix lookup ```filter(meta={})``` 16 | 17 | 2017-08-31: 0.0.27 18 | Fix as_{} lookup for python3 19 | 20 | 2017-08-31: 0.0.26 21 | Fix has lookup after Django 1.10 22 | 23 | 2017-05-18：0.0.25 24 | Supress exception when drop index and the index already removed. 25 | 26 | 2017-03-14: 0.0.24 27 | Add support for __near lookup with postgres earthdistance plugin, Thanks to @steinliber 28 | 29 | 2016-06-01: 0.0.23 30 | Fix value from select_json not been decode from json introduce by 0.0.18 31 | 32 | 2016-03-24: 0.0.22 33 | Fix error #11 remove the unexpect decode float to Decimal 34 | 35 | 2016-03-19: 0.0.21 36 | Fix error #10 37 | 38 | 2016-03-09: 0.0.20 39 | Add the array length for select_json 40 | 41 | 2016-03-08: 0.0.19 42 | fix when add a json field with db_index=True and it's fail to generate the create index sql 43 | 44 | 2016-03-01: 0.0.18 45 | we want to be able to use customize decoder to load json, so get avoid the psycopg2's decode json, just return raw text then we deserilize by the field from_db_value 46 | 47 | 2016-03-01: 0.0.17 48 | patch the django serilizer to not return the stringifyed result 49 | 50 | 2015-07-23: 0.0.16 51 | Add support for ./manage.py inspectdb 52 | 53 | 2015-06-10: 0.0.15 54 | Add support for db_index to add GIN index 55 | 56 | Install 57 | ======= 58 | 59 | `pip install django-pgjsonb` 60 | 61 | 62 | Definition 63 | === 64 | 65 | ```python 66 | from django_pgjsonb import JSONField 67 | 68 | class Article(models.Model): 69 | meta=JSONField([null=True,default={},decode_kwargs={},encode_kwargs={},db_index=False,db_index_options={}]) 70 | ``` 71 | 72 | 73 | Encoder and Decoder Options 74 | === 75 | by define decode_kwargs and encode_kwargs you can use your customize json dump and load behaveior, basicly these parameters will just pass to json.loads(**decode_kwargs) and json.dumps(**encode_kwargs) 76 | 77 | here is an example for use [EJSON](https://pypi.python.org/pypi/ejson) to store native datetime object 78 | 79 | ```python 80 | import ejson 81 | 82 | class Article(models.Model): 83 | meta=JSONField(encode_kwargs={"cls":ejson.EJSONEncoder},decode_kwargs={"cls":ejson.EJSONDecoder}) 84 | ``` 85 | 86 | 87 | Add Index 88 | ===== 89 | [new add in 0.0.15] 90 | 91 | jsonb field support gin type index to accelerator filtering. Since JSON is a data structure contains hierarchy, so the index of jsonb field will be more complicate than another single value field. More information, please reference [Postgres document 8.14.4](http://www.postgresql.org/docs/9.4/static/datatype-json.html) 92 | 93 | ```python 94 | meta=JSONField(db_index=True) 95 | or 96 | meta=JSONField(db_index=True,db_index_options={"path":"authors__name","only_contains":True}) 97 | or 98 | meta=JSONField(db_index=True,db_index_options=[{},{"path":"authors__name","only_contains":True}]) 99 | ``` 100 | 101 | When set db_index as True and do not set db_index_options, it will generate default GIN index, most case it's enough. 102 | 103 | When specify ```db_index_options={"only_contains":True}```, the index will be as the non-default GIN operator class jsonb_path_ops that supports indexing the ```contains``` operator only, but it's consume less space and more efficient. 104 | 105 | When specify the path parameter in db_index_options, ```db_index_options={"path":"authors__name"}```, then index will generate to the specify path, so that ```Article.objects.filter(meta__authors__name__contains=["asd"])``` can utilize the index. 106 | 107 | So you can create multiple index in one JSONField, just pass the db_index_options parameter as a list that contains multiple options, it will generate multiple correspond indexes. Empty dict stand for the default GIN index. 108 | 109 | 110 | Lookups 111 | ======= 112 | ###Contains a wide range of lookups supported natively by postgres 113 | 114 | 1. `has` :if field has specific key *`("?")`* 115 | 116 | ```python 117 | Article.objects.filter(meta__has="author") 118 | ``` 119 | 120 | 2. `has_any` : if field has any of the specific keys *`("?|")`* 121 | 122 | ```python 123 | Article.objects.filter(meta__has_any=["author","date"]) 124 | ``` 125 | 3. `has_all` : if field has all of the specific keys *`("?&")`* 126 | 127 | ```python 128 | Article.objects.filter(meta__has_all=["author","date"]) 129 | ``` 130 | 4. `contains` : if field contains the specific keys and values *`("@>")`* 131 | ```python 132 | Article.objects.filter(meta__contains={"author":"yjmade","date":"2014-12-13"}) 133 | ``` 134 | 135 | 5. `in` or `contained_by` : if all field key and value contain by input *`("<@")`* 136 | ```python 137 | Article.objects.filter(meta__in={"author":"yjmade","date":"2014-12-13"}) 138 | ``` 139 | 140 | 6. `len` : the length of the array, transform to int, and can followed int lookup like gt or lt *`("jsonb_array_length()")`* 141 | 142 | ```python 143 | Article.objects.filter(meta__authors__len__gte=3) 144 | Article.objects.filter(meta__authors__len=10) 145 | ``` 146 | 7. `as_(text,int,float,bool,date,datetime)` : transform json field into specific data type so that you can follow operation of this type *`("CAST(FIELD as TYPE)")`* 147 | 148 | ```python 149 | Article.objects.filter(meta__date__as_datetime__year__range=(2012,2015)) 150 | Article.objects.filter(meta__view_count__as_float__gt=100) 151 | Article.objects.filter(meta__title__as_text__iregex=r"^\d{4}") 152 | ``` 153 | 8. `path_(PATH)` : get the specific path, path split by '_' *`("#>")`* 154 | 155 | ```python 156 | Article.objects.filter(meta__path_author_articles__contains="show me the money") 157 | ``` 158 | 159 | 160 | Extend function to QuerySet 161 | ======================== 162 | 1.`select_json("JSON_PATHS",field_name="JSON_PATHS")` 163 | 164 | JSON_PATHS in the format of paths separated by "__",like "meta__location__geo_info". It will use the queryset's `extra` method to transform a value inside json as a field. 165 | If no field_name provided, it will generate a field name with lookups separate by _ without the json field self's name, so `select_json("meta__author__name")` is equal to `select_json(author_name="meta__author__name")` 166 | 167 | ```python 168 | Article.objects.select_json("meta__author__name",geo="meta__location__geo_info")` 169 | ``` 170 | 171 | This operation will translate to sql as 172 | 173 | ```sql 174 | SELECT "article"."meta"->'location'->'geo_info' as "geo", "article"."meta"->'author'->'name' as "author_name" 175 | ``` 176 | 177 | [new add in 0.0.20] 178 | You can also select the length of a json array as a field by use Length object 179 | 180 | ```python 181 | from django_pgjsonb.fields import Length 182 | Article.objects.select_json(authors_len=Length("meta__authors")).values("authors_len") 183 | ``` 184 | 185 | After select_json, the field_name can be operate in values() and values_list() method, so that 186 | 187 | 1. select only one specific value inside json 188 | 2. to group by one value inside json 189 | 190 | is possible. 191 | 192 | Demo: 193 | 194 | ```python 195 | Article.objects.all().select_json(tags="meta__tags").values_list("tags") 196 | # select only "meta"->'tags' 197 | 198 | Article.objects.all().select_json(author_name="meta__author__name")\ 199 | .values("author_name").annotate(count=models.Count("author_name")) 200 | # GROUP BY "meta"->'author'->'name' 201 | ``` 202 | 203 | 204 | 205 | 206 | support geo search in jsonb 207 | =========================== 208 | 209 | **require**: postgresql plugin: 210 | 211 | 1. cube 212 | 213 | 2. earthdistance 214 | 215 | 3. to install these two plugin, run command below in psql 216 | 217 | ``` 218 | CREATE EXTENSION cube; 219 | CREATE EXTENSION earthdistance; 220 | ``` 221 | 222 | how to save location json record 223 | 224 | ```Json 225 | {"location": [30.2, 199.4]} # just keep a latitude, longitude list 226 | ``` 227 | 228 | Demo 229 | 230 | ```python 231 | Article.objects.filter(data__location__near=[39.9, 116.4,5000]) # latitude，longitude，search range 232 | ``` 233 | 234 | or 235 | 236 | ```python 237 | Article.objects.filter(data__location__near='39.9,116.4,5000') # latitude，longitude, search range 238 | ``` 239 | 240 | **Alert**: if you don't pass exact number of params, this filter will not be used 241 | 242 | **for more earthdistance**, see [Postgresql Earthdistance Documentation](https://www.postgresql.org/docs/8.3/static/earthdistance.html) 243 | 244 | ------------------------------------------------------------------------------------------------------------------ 245 | 246 | 247 | #####For more information about raw jsonb operation, please see [PostgreSQL Documentation](http://www.postgresql.org/docs/9.4/static/functions-json.html) -------------------------------------------------------------------------------- /django_pgjsonb/__init__.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | from .fields import * # noqa 3 | -------------------------------------------------------------------------------- /django_pgjsonb/fields.py: -------------------------------------------------------------------------------- 1 | from __future__ import unicode_literals 2 | 3 | import json 4 | import logging 5 | 6 | from django.core.serializers.json import DjangoJSONEncoder 7 | from django.db.utils import ProgrammingError, InternalError 8 | from django.db import models 9 | from django.db.models.lookups import BuiltinLookup, Transform 10 | from django.db.backends.postgresql_psycopg2.schema import DatabaseSchemaEditor 11 | from django.db.backends.postgresql_psycopg2.introspection import DatabaseIntrospection 12 | from django.utils import six 13 | from psycopg2.extras import register_default_jsonb, Json 14 | # we want to be able to use customize decoder to load json, so get avoid the psycopg2's decode json, just return raw text then we deserilize by the field from_db_value 15 | logger = logging.getLogger(__name__) 16 | register_default_jsonb(loads=lambda x: x) 17 | 18 | DatabaseIntrospection.data_types_reverse[3802] = "django_pgjsonb.JSONField" 19 | 20 | 21 | class JsonAdapter(Json): 22 | """ 23 | Customized psycopg2.extras.Json to allow for a custom encoder. 24 | """ 25 | 26 | def __init__(self, adapted, dumps=None, encode_kwargs=None): 27 | self.encode_kwargs = encode_kwargs 28 | super(JsonAdapter, self).__init__(adapted, dumps=dumps) 29 | 30 | def dumps(self, obj): 31 | # options = {'cls': self.encoder} if self.encoder else {} 32 | return json.dumps(obj, **self.encode_kwargs) 33 | 34 | 35 | class JSONField(models.Field): 36 | description = 'JSON Field' 37 | 38 | def __init__(self, *args, **kwargs): 39 | self.decode_kwargs = kwargs.pop('decode_kwargs', { 40 | # 'parse_float': decimal.Decimal 41 | }) 42 | self.encode_kwargs = kwargs.pop('encode_kwargs', { 43 | 'cls': DjangoJSONEncoder, 44 | }) 45 | db_index = kwargs.get("db_index") 46 | db_index_options = kwargs.pop("db_index_options", {}) 47 | if db_index: 48 | self.db_index_options = db_index_options if isinstance(db_index_options, (list, tuple)) else [db_index_options] 49 | 50 | kwargs["db_index"] = False # to supress the system default create_index_sql 51 | super(JSONField, self).__init__(*args, **kwargs) 52 | 53 | def get_internal_type(self): 54 | return 'JSONField' 55 | 56 | def db_type(self, connection): 57 | return 'jsonb' 58 | 59 | def get_prep_value(self, value): 60 | if value is None: 61 | return None 62 | return JsonAdapter(value, encode_kwargs=self.encode_kwargs) 63 | 64 | def get_prep_lookup(self, lookup_type, value, prepared=False): 65 | if lookup_type == 'has': 66 | # Need to ensure we have a string, as no other 67 | # values is appropriate. 68 | if not isinstance(value, six.string_types): 69 | value = '%s' % value 70 | if lookup_type in ['has_all', 'has_any']: 71 | # This lookup type needs a list of strings. 72 | if isinstance(value, six.string_types): 73 | value = [value] 74 | # This will cast numbers to strings, but also grab the keys 75 | # from a dict. 76 | value = ['%s' % v for v in value] 77 | if lookup_type == 'near': 78 | # geo type must have 3 item, longitude, latitude and search 79 | # range, allowed_format 'lat,lng,rang' or [lat, lng, rang] 80 | if isinstance(value, six.string_types): 81 | value = value.split(',') 82 | # if parmas not verify, just don't use this filter, 12756000 83 | # is the farest long in earth 84 | if len(value) != 3: 85 | value = [0, 0, 12756000] 86 | return value 87 | 88 | def get_db_prep_lookup(self, lookup_type, value, connection, prepared=False): 89 | if lookup_type in ['contains', 'in']: 90 | value = self.get_prep_value(value) 91 | return [value] 92 | 93 | return super(JSONField, self).get_db_prep_lookup(lookup_type, value, connection, prepared) 94 | 95 | def deconstruct(self): 96 | name, path, args, kwargs = super(JSONField, self).deconstruct() 97 | path = 'django_pgjsonb.fields.JSONField' 98 | kwargs.update( 99 | decode_kwargs=self.decode_kwargs, 100 | encode_kwargs=self.encode_kwargs 101 | ) 102 | if hasattr(self, "db_index_options"): 103 | if self.db_index_options: 104 | kwargs["db_index_options"] = self.db_index_options 105 | kwargs["db_index"] = True 106 | return name, path, args, kwargs 107 | 108 | def to_python(self, value): 109 | if value is None and not self.null and self.blank: 110 | return '' 111 | # Rely on psycopg2 to give us the value already converted. 112 | return value 113 | 114 | def from_db_value(self, value, expression, connection, context): 115 | if value is not None: 116 | value = json.loads(value, **self.decode_kwargs) 117 | return value 118 | 119 | def get_transform(self, name): 120 | transform = super(JSONField, self).get_transform(name) 121 | if transform: 122 | return transform 123 | 124 | if name.startswith('path_'): 125 | path = '{%s}' % ','.join(name.replace('path_', '').split('_')) 126 | return PathTransformFactory(path) 127 | 128 | try: 129 | name = int(name) 130 | except ValueError: 131 | pass 132 | 133 | return GetTransform(name) 134 | 135 | 136 | def patch_index_create(): 137 | DatabaseSchemaEditor.create_jsonb_index_sql = "CREATE INDEX %(name)s ON %(table)s USING GIN ({path}{ops_cls})%(extra)s" 138 | 139 | def get_jsonb_index_name(editor, model, field, index_info): 140 | return editor._create_index_name(model, [field.name], suffix=editor._digest(index_info.get("path") or "")) 141 | 142 | def create_jsonb_index_sql(editor, model, field): 143 | options = field.db_index_options 144 | 145 | json_path_op = "->" 146 | sqls = {} 147 | for option in options: 148 | paths = option.get("path", "") 149 | if not paths: 150 | path = "%(columns)s" 151 | else: 152 | path_elements = paths.split("__") 153 | path = "(%(columns)s{}{})".format(json_path_op, json_path_op.join(["'%s'" % element for element in path_elements])) 154 | 155 | ops_cls = " jsonb_path_ops" if option.get("only_contains") else "" 156 | sql = editor.create_jsonb_index_sql.format(path=path, ops_cls=ops_cls) 157 | sqls[get_jsonb_index_name(editor, model, field, option)] = editor._create_index_sql(model, [field], sql=sql, suffix=(editor._digest(paths) if paths else "")) 158 | return sqls 159 | 160 | DatabaseSchemaEditor._create_jsonb_index_sql = create_jsonb_index_sql 161 | 162 | orig_model_indexes_sql = DatabaseSchemaEditor._model_indexes_sql 163 | orig_alter_field = DatabaseSchemaEditor._alter_field 164 | orig_add_field = DatabaseSchemaEditor.add_field 165 | 166 | def _model_indexes_sql(editor, model): 167 | output = orig_model_indexes_sql(editor, model) 168 | json_fields = [field for field in model._meta.local_fields if isinstance(field, JSONField) and hasattr(field, "db_index_options")] 169 | for json_field in json_fields: 170 | output.extend(editor._create_jsonb_index_sql(model, json_field).values()) 171 | return output 172 | 173 | def _alter_field(editor, model, old_field, new_field, old_type, new_type, 174 | old_db_params, new_db_params, strict=False): 175 | res = orig_alter_field(editor, model, old_field, new_field, old_type, new_type, 176 | old_db_params, new_db_params, strict=False) 177 | if not isinstance(new_field, JSONField): 178 | return res 179 | old_index = getattr(old_field, "db_index_options", None) 180 | new_index = getattr(new_field, "db_index_options", None) 181 | if new_index != old_index: 182 | all_old_index_names = {get_jsonb_index_name(editor, model, new_field, index_info) for index_info in old_index} if old_index else set() 183 | all_new_indexe_names = {get_jsonb_index_name(editor, model, new_field, index_info) for index_info in new_index} if new_index else set() 184 | 185 | to_create_indexs, to_delete_indexes = (all_new_indexe_names - all_old_index_names), (all_old_index_names - all_new_indexe_names) 186 | if to_create_indexs: 187 | for index_name, sql in six.iteritems(editor._create_jsonb_index_sql(model, new_field)): 188 | if index_name in to_create_indexs: 189 | editor.execute(sql) 190 | 191 | for index_name in to_delete_indexes: 192 | try: 193 | editor.execute(editor._delete_constraint_sql(editor.sql_delete_index, model, index_name)) 194 | except (ProgrammingError, InternalError) as exc: 195 | logger.warning(exc) 196 | continue 197 | return res 198 | 199 | def add_field(editor, model, field): 200 | res = orig_add_field(editor, model, field) 201 | if not isinstance(field, JSONField): 202 | return res 203 | if getattr(field, "db_index_options", None): 204 | editor.deferred_sql.extend(editor._create_jsonb_index_sql(model, field).values()) 205 | return res 206 | 207 | DatabaseSchemaEditor._model_indexes_sql = _model_indexes_sql 208 | DatabaseSchemaEditor._alter_field = _alter_field 209 | DatabaseSchemaEditor.add_field = add_field 210 | 211 | 212 | patch_index_create() 213 | 214 | 215 | class PostgresLookup(BuiltinLookup): 216 | 217 | def process_lhs(self, qn, connection, lhs=None): 218 | lhs = lhs or self.lhs 219 | return qn.compile(lhs) 220 | 221 | def get_rhs_op(self, connection, rhs): 222 | return '%s %s' % (self.operator, rhs) 223 | 224 | 225 | class EarthNearLookup(BuiltinLookup): 226 | ''' 227 | Eeed plugin: cube and earthdistance, This class help build geo 228 | search sql by earthdistance func 229 | ''' 230 | 231 | def process_lhs(self, qn, connection, lhs=None): 232 | lhs = lhs or self.lhs 233 | lhs_value, params = qn.compile(lhs) 234 | earth_lhs_value = 'll_to_earth((({0})::json->>0)::float,' \ 235 | '(({0})::json->>1)::float)'.format(lhs_value) 236 | return earth_lhs_value, params 237 | 238 | def get_rhs_op(self, connection, rhs): 239 | return '{0} earth_box(ll_to_earth({1},{1}), {1})'.format(self.operator, rhs) 240 | 241 | def as_sql(self, compiler, connection): 242 | lhs_sql, params = self.process_lhs(compiler, connection) 243 | rhs_sql, rhs_params = self.process_rhs(compiler, connection) 244 | rhs_params = [item for sublist in rhs_params 245 | for item in sublist] 246 | params.extend(rhs_params) 247 | rhs_sql = self.get_rhs_op(connection, rhs_sql) 248 | return '%s %s' % (lhs_sql, rhs_sql), params 249 | 250 | 251 | class Near(EarthNearLookup): 252 | lookup_name = 'near' 253 | operator = '<@' 254 | 255 | 256 | JSONField.register_lookup(Near) 257 | 258 | 259 | class Has(PostgresLookup): 260 | lookup_name = 'has' 261 | operator = '?' 262 | prepare_rhs = False 263 | 264 | 265 | JSONField.register_lookup(Has) 266 | 267 | 268 | class Contains(PostgresLookup): 269 | lookup_name = 'contains' 270 | operator = '@>' 271 | 272 | 273 | JSONField.register_lookup(Contains) 274 | 275 | 276 | class ContainedBy(PostgresLookup): 277 | lookup_name = 'contained_by' 278 | operator = '<@' 279 | 280 | def as_sql(self, compiler, connection): 281 | lhs_sql, params = self.process_lhs(compiler, connection) 282 | rhs_sql, rhs_params = self.process_rhs(compiler, connection) 283 | params.extend(rhs_params) 284 | rhs_sql = self.get_rhs_op(connection, rhs_sql) 285 | return "%s!='{}' and %s %s" % (lhs_sql, lhs_sql, rhs_sql), params 286 | 287 | 288 | JSONField.register_lookup(ContainedBy) 289 | 290 | 291 | class In(ContainedBy): 292 | lookup_name = 'in' 293 | 294 | 295 | JSONField.register_lookup(In) 296 | 297 | 298 | class HasAll(PostgresLookup): 299 | lookup_name = 'has_all' 300 | operator = '?&' 301 | prepare_rhs = False 302 | 303 | 304 | JSONField.register_lookup(HasAll) 305 | 306 | 307 | class HasAny(PostgresLookup): 308 | lookup_name = 'has_any' 309 | operator = '?|' 310 | prepare_rhs = False 311 | 312 | 313 | JSONField.register_lookup(HasAny) 314 | 315 | 316 | # class ArrayLength(BuiltinLookup): 317 | # lookup_name = 'array_length' 318 | 319 | # def as_sql(self, qn, connection): 320 | # lhs, lhs_params = self.process_lhs(qn, connection) 321 | # rhs, rhs_params = self.process_rhs(qn, connection) 322 | # params = lhs_params + rhs_params 323 | # return 'jsonb_array_length(%s) = %s' % (lhs, rhs), params 324 | 325 | # JSONField.register_lookup(ArrayLength) 326 | 327 | class ArrayLenTransform(Transform): 328 | lookup_name = 'len' 329 | 330 | @property 331 | def output_field(self): 332 | return models.IntegerField() 333 | 334 | def as_sql(self, qn, connection): 335 | lhs, params = qn.compile(self.lhs) 336 | return 'jsonb_array_length(%s)' % lhs, params 337 | 338 | 339 | JSONField.register_lookup(ArrayLenTransform) 340 | 341 | 342 | class TransformMeta(type(Transform)): 343 | 344 | def __init__(cls, *args): # noqa 345 | super(TransformMeta, cls).__init__(*args) 346 | cls.lookup_name = "as_%s" % (cls.lookup_type or cls.type) 347 | 348 | if cls.__name__ != "AsTransform": 349 | JSONField.register_lookup(cls) 350 | 351 | 352 | class AsTransform(six.with_metaclass(TransformMeta, Transform)): 353 | type = None 354 | lookup_type = None 355 | field_type = None 356 | 357 | def as_sql(self, qn, connection): 358 | lhs, params = qn.compile(self.lhs) 359 | splited = lhs.split("->") 360 | lhs = "->>".join(["->".join(splited[:-1]), splited[-1]]) 361 | return "CAST(%s as %s)" % (lhs, self.type), params 362 | 363 | @property 364 | def output_field(self): 365 | return self.field_type() 366 | 367 | 368 | class JsonAsText(AsTransform): 369 | type = "text" 370 | field_type = models.CharField 371 | 372 | 373 | class JsonAsInteger(AsTransform): 374 | type = "integer" 375 | field_type = models.IntegerField 376 | lookup_type = "int" 377 | 378 | 379 | class JsonAsFloat(AsTransform): 380 | type = "float" 381 | field_type = models.FloatField 382 | 383 | 384 | class JsonAsBool(AsTransform): 385 | type = "boolean" 386 | field_type = models.NullBooleanField 387 | lookup_type = "bool" 388 | 389 | 390 | class JsonAsDate(AsTransform): 391 | type = "date" 392 | field_type = models.DateField 393 | 394 | 395 | class JsonAsDatetime(AsTransform): 396 | lookup_type = "datetime" 397 | field_type = models.DateTimeField 398 | 399 | @property 400 | def type(self): 401 | from django.conf import settings 402 | if settings.USE_TZ: 403 | return "timestamptz" 404 | else: 405 | return "timestamp" 406 | 407 | 408 | class Get(Transform): 409 | 410 | def __init__(self, name, *args, **kwargs): 411 | super(Get, self).__init__(*args, **kwargs) 412 | self.name = name 413 | 414 | def as_sql(self, qn, connection): 415 | lhs, params = qn.compile(self.lhs) 416 | # So we can run a query of this type against a column that contains 417 | # both array-based and object-based (and possibly scalar) values, 418 | # we need to add an additional WHERE clause that ensures we only 419 | # get json objects/arrays, as per the input type. 420 | # It would be really nice to be able to see if these clauses 421 | # have already been applied. 422 | 423 | # if isinstance(self.name, six.string_types): 424 | # Also filter on objects. 425 | # filter_to = "%s @> '{}' AND" % lhs 426 | # self.name = "'%s'" % self.name 427 | # elif isinstance(self.name, int): 428 | # Also filter on arrays. 429 | # filter_to = "%s @> '[]' AND" % lhs 430 | if isinstance(self.name, int): 431 | return '%s -> %s' % (lhs, self.name), params 432 | return '%s -> \'%s\'' % (lhs, self.name), params 433 | 434 | 435 | class GetTransform(object): 436 | 437 | def __init__(self, name): 438 | self.name = name 439 | 440 | def __call__(self, *args, **kwargs): 441 | return Get(self.name, *args, **kwargs) 442 | 443 | 444 | class Path(Transform): 445 | 446 | def __init__(self, path, *args, **kwargs): 447 | super(Path, self).__init__(*args, **kwargs) 448 | self.path = path 449 | 450 | def as_sql(self, qn, connection): 451 | lhs, params = qn.compile(self.lhs) 452 | # Because path operations only work on non-scalar types, we 453 | # need to filter out scalar types as part of the query. 454 | return "({0} @> '[]' OR {0} @> '{{}}') AND {0} #> '{1}'".format(lhs, self.path), params 455 | 456 | 457 | class PathTransformFactory(object): 458 | 459 | def __init__(self, path): 460 | self.path = path 461 | 462 | def __call__(self, *args, **kwargs): 463 | return Path(self.path, *args, **kwargs) 464 | 465 | 466 | def patch_select_json(): 467 | class SQLStr(six.text_type): 468 | output_field = JSONField() 469 | 470 | def select_json(query, *args, **kwargs): 471 | if not args and not kwargs: 472 | return query 473 | 474 | def get_sql_str(model, opr): 475 | if isinstance(opr, six.string_types): 476 | opr_elements = opr.split("__") 477 | field = opr_elements.pop(0) 478 | select_elements = ['"%s"."%s"' % (model._meta.db_table, field)] + [("%s" if name.isnumeric() else "'%s'") % name for name in opr_elements] 479 | return "_".join(opr_elements), SQLStr(" -> ".join(select_elements)) 480 | elif isinstance(opr, Length): 481 | annotate_name, sql = get_sql_str(model, opr.field_select) 482 | return annotate_name + "_len", SQLStr("jsonb_array_length(%s)" % sql) 483 | 484 | return query.extra(select=dict( 485 | [get_sql_str(query.model, opr) for opr in args], 486 | **{k: get_sql_str(query.model, v)[1] for k, v in six.iteritems(kwargs)} 487 | )) 488 | 489 | models.QuerySet.select_json = select_json 490 | 491 | orig_init = models.expressions.RawSQL.__init__ 492 | 493 | def init(self, sql, params, output_field=None): 494 | if hasattr(sql, "output_field"): 495 | output_field = sql.output_field 496 | return orig_init(self, sql, params, output_field=output_field) 497 | 498 | models.expressions.RawSQL.__init__ = init 499 | 500 | 501 | patch_select_json() 502 | 503 | 504 | class Length(object): 505 | 506 | def __init__(self, field_select): 507 | self.field_select = field_select 508 | 509 | 510 | def manager_select_json(manager, *args, **kwargs): 511 | return manager.all().select_json(*args, **kwargs) 512 | 513 | 514 | models.manager.BaseManager.select_json = manager_select_json 515 | 516 | 517 | def patch_serializer(): 518 | from django.core.serializers.python import Serializer 519 | old_handle_field = Serializer.handle_field 520 | 521 | def handle_field(serializer, obj, field): 522 | if isinstance(field, JSONField): 523 | value = field.value_from_object(obj) 524 | serializer._current[field.name] = value 525 | else: 526 | return old_handle_field(serializer, obj, field) 527 | 528 | Serializer.handle_field = handle_field 529 | 530 | 531 | patch_serializer() 532 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [metadata] 2 | description-file = README.md 3 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | from setuptools import setup, find_packages 4 | 5 | try: 6 | README = open('README.md').read() 7 | except UnicodeDecodeError: 8 | README = "" 9 | 10 | VERSION = "0.0.31" 11 | 12 | setup( 13 | name='django-pgjsonb', 14 | version=VERSION, 15 | description='Django Postgres JSONB Fields support with lookups', 16 | url="https://github.com/yjmade/django-pgjsonb", 17 | long_description=README, 18 | author='Jay Young(yjmade)', 19 | author_email='dev@yjmade.net', 20 | packages=find_packages(), 21 | install_requires=['Django>=1.7'], 22 | ) 23 | --------------------------------------------------------------------------------