Thanks for contacting for this feature.

There is a huge demand to use http port while using spring data elasticsearch library either stand alone or with spring boot.

We would love to have JEST implementation in spring data elasticsearch.

Please fill free to open a pull request.

Ok thanks, i will be starting to work on this PR.

Do you have any whishes about ES version compatibility ?

Indeed Jest maintain two version :

1.X.X : Compatible with ES 1.7
2.X.X : Compatible with ES 2.0 ( and i don't know if it's working with lesser than ES 2.0 )

We are actively working to support 2.x version. better to target 2.x as of now.

Hi,

My first implementation is visible here :
https://github.com/VanRoy/spring-data-elasticsearch/tree/jest-implementation-2.0

It's based on your branch : issue/DATAES-211.
It pass 50 of 65 tests of ElasticsearchTemplateTests.

Currently it's a very conservative implementation ( no changes on existing classes ).

I will work on another implementation with more factorization of code between ElasticsearchTemplate and my JestElasticsearchTemplate.

I thinking on classes tree like that :

• ElasticsearchOperations ( base interface that contains common methods )
  -> JestElasticsearchOperations ( Specific methods for Jest backend )
  -> NativeElasticsearchOperations ( Specific methods for Native ES )

• ElasticsearchTemplate ( base class that contains common code )
  -> JestElasticsearchTemplate ( Specific implementation for Jest backend )
  -> NativeElasticsearchTemplate ( Specific implementation for Native ES )

What do you think about that ?

Thanks for you help.
Julien.

+1 On this.. I'm not sure why it's a minor. For me this is a blocker.

Petar, i agree with you.

Mohsin, I would like to complete my work about this feature.
Can you tell me what do you think about my previous message ( refactorization or conservative implementation ).

Thanks a lot.
Julien.

Thanks Julien

We will review the PR soon. and we have working branch for 2.x now.

https://github.com/spring-projects/spring-data-elasticsearch/tree/DATAES-211-ES2.0

After talking to various Spring Data devs about this feature, all have one common suggestion

To have separate ElasticsearchTemplate for Jest is ok, as you need to have java code which deals with http calls using http client.

but to add one more mapper for GSON can be problematic, as Spring Data elasticsearch already have jackson for mapping in and out from elasticsearch.

Also adding additional third party library we will be bound to releases of other libraries too.

Thanks Julien, this looks great! We would like to test the jest support PR, is there
a way to do this without getting the source?

To add above, i have created a branch

https://github.com/spring-projects/spring-data-elasticsearch/tree/DATAES-220

please propose a PR to this and we will accept and add to build pipeline so that we have stable version where end user can test the feature.

I am very interested in JEST implementation merged.
Is it still planned for Hopper RC1 release, 1st week of March?

Julien, should the branch in your fork compile without errors? We're trying to push it in our private repo in the meanwhile for testing, but see some compilation errors, mostly regarding `scroll`:

"(cannot infer type-variable(s) T
(argument mismatch; <anonymous org.springframework.data.elasticsearch.core.JestSearchResultMapper> cannot be converted to org.springframework.data.elasticsearch.core.SearchResultMapper))
method org.springframework.data.elasticsearch.core.JestElasticsearchTemplate.<T>scroll(java.lang.String,long,org.springframework.data.elasticsearch.core.JestScrollResultMapper) is not applicable"

not possible to add it in next release(Hopper RC1), as we still don't have working branch.

i've been looking into getting this branch to work here https://github.com/pulse00/spring-data-elasticsearch/tree/jest-implementation-2.0 .

I'm not sure if the original author has still time to finish this PR - if not our company would like to offer help implementing the currently unimplemented parts (mainly faceted search what i've seen and some scroll-mapping issues).

Julien, would you be ok with this?

Robert,

Indeed, I did not have the time to work on this feature this weeks.
I will be very happy to share the work with other people.

At this times, two question will be on hold :

1) Did you make an abstract ElasticsearchTemplate for share code between NativeElasticsearchTemplate and JestElasticsearchTemplate or just duplicate like i have doing in the current implementation ?

2) The Jest client library come with GSON dependency and Spring Data ES use Jackson. How solve this problem ?
By contributing to current experimentation on Jest that move from GSON to Jackson (https://github.com/ctrimble/Jest/tree/experiment_object-mapper)
or
By remove Jest and re-implementing all HTTP calls.

I waiting for responses from Mohsin about this questions and continue my work after.
Julien.

Regarding the ElasticsearchTemplate: I think we might opt for a similar approach like spring-data-redis. There you have the option of multiple underlying clients without having to change the template:

The main concepts there involve connection factory and a connection interface:

• https://github.com/spring-projects/spring-data-redis/blob/master/src/main/java/org/springframework/data/redis/connection/RedisConnectionFactory.java
• https://github.com/spring-projects/spring-data-redis/blob/master/src/main/java/org/springframework/data/redis/connection/RedisConnection.java

Maybe this is a viable approach?

Regarding the gson/jackson issue: Re-Implementing jest doesn't sound very appealing to me, so maybe we can get in touch with the jest people and check the status of the gson refactoring?

-robert

One of the reason why I moved to Spring Elastic Search was because it used Jackson for JSON parsing. Looks like GSON won't leave me alone
Re Jest moving to Jackson, this has been ongoing for a while now (https://github.com/searchbox-io/Jest/issues/45). I don't think it is happening anytime soon, and I rather have Spring continue its use of Jackson. Not sure if the rest of the community agrees with me..

Any news here? What is the bottleneck of merging this?

I will have time this week to work on the pull request. Fyi: I'll use jest as-is (meaning with the jackson dependency).

I have merged the latest spring-data-elasticsearch master branch into the jest fork, changes can be found here: https://github.com/pulse00/spring-data-elasticsearch/tree/feature/jest

To finish this there are 2 options:

Option #1: Keep a single implementation of ElasticsearchTemplate working with both the elasticsearch and the jest client

This option would require some substantial refactoring to be done, most likely with breaking backwards compatibility.
The reason is that ElasticSearchOperations and the classes used in the interface signatures are pretty much tied to
the elasticsearch client implementation.

Here's a couple of examples:

• https://github.com/spring-projects/spring-data-elasticsearch/blob/master/src/main/java/org/springframework/data/elasticsearch/core/ElasticsearchOperations.java#L18-L20
• https://github.com/spring-projects/spring-data-elasticsearch/blob/master/src/main/java/org/springframework/data/elasticsearch/core/SearchResultMapper.java#L18

Especially the *ResultMapper classes rely on the ES client API.

If introducing breaking changes is ok for the core committers, i can work on a refactoring to

• Remove ES specific API from the ElasticsearchOperations interface
• Refactor the Mapper and Search classes to remove the ES specific api

Option #2: Keep the current implementation (2 separate implementations of ElasticsearchOperations)

There would be 2 implementations:

1. The existing ElasticSearchTemplate
2. The new JestElasticSearchTemplate

Existing project would not require any changes, but the new JestElasticSearchTemplate would have some methods
throw an UnsupportedOperationException because the interface signature is incompatible with the jest mapping / search results.

Examples:

• https://github.com/pulse00/spring-data-elasticsearch/blob/feature/jest/src/main/java/org/springframework/data/elasticsearch/core/JestElasticsearchTemplate.java#L298
• https://github.com/pulse00/spring-data-elasticsearch/blob/feature/jest/src/main/java/org/springframework/data/elasticsearch/core/JestElasticsearchTemplate.java#L786

Any feedback on these 2 options would be greatly appreciated.

Here's my pull request with option #2 from the above comment.

https://github.com/spring-projects/spring-data-elasticsearch/pull/144

This commit requires the next patchlevel version of jest though, because of https://github.com/searchbox-io/Jest/pull/340.

Hi Moshin,

Do you see any inconveniences to I package my JestElasticSearchTemplate in new and independent Github projet and in the Maven Central ?

You can see the projet in this Github repository : https://github.com/VanRoy/spring-data-jest
I will waiting for your agreement to publish to maven central.

Thanks.
Julien.

Hi Moshin,

Finally, I have published the artefact on maven central.

Do not hesitate to come back to me if any problems related to this publicated.

Thank a lot.
Julien.

Is the Jest implementation the only planned implementation for talking to ES via Http? ES have released (alpha) of a RestClient. Will this be used as an alternative to Jest?

I read a notice on spring blog that there i now support for Jest . Does it mean this has been merged to a release ? Which one , because the most current on spring repo is still the old spring-data-elasticsearch 1.1.0

https://spring.io/blog/2016/07/12/this-week-in-spring-july-12th-2016

> The first release candidate of Spring Boot 1.4 is now available, packed with a lot of great things! This new release features, among other things, a unified @EntityScan for JPA, MongoDB, Neo4j, Couchbase and Cassandra, an auto-configured RestTemplateBuilder, support for pure REST client tests via @RestClientTest, support for Jest (the Elasticsearch REST client), and upgrades to Spring Integration 4.3, Spring AMQP 1.6, Spring REST Docs 1.1, MongoDB Java Driver 3 and more! Download the bits and kick the tires right now!

@Terence, blog you are referring to is the news about Spring Boot supporting Jest client, not in the actual spring data elasticsearch.
You can get more info at https://github.com/spring-projects/spring-boot/issues/6032

@Amin : Yes once elasticsearch 5.x is released we are planning to support Native Rest client.

@Mohsin: Are you planning to release Julien's Jest implementation in the meantime while waiting for Elasticsearch 5.x? Or is it being scrapped in favor of moving directly to the native REST client? Just wondering what all of this means in terms of how long we'll need to wait before this feature is actually usable in production.

Also, would this driver be compatible with the AWS Elasticsearch Service, or does Amazon need to support 5.x first?

@Joseph In waiting of decision, you can already use my implementation ( https://github.com/VanRoy/spring-data-jest ).
It's available on maven central, works well on AWS with ES 1.5 and 2.3.
It's compatible with Spring Boot 1.3, 1.4 and Spring Data 1.3 and 2.0.

Julien.

@Julien what is the status of this? Is it planned to be integrate any time soon? btw I tried to use your existing implementation but there are (small) issues (already pointed out on Github) that prevent a correct functionality using the latest elastic search. When do you think you will release your library? Thanks

Hi @Fabio,

In my opinion, I think that it will never be integrated in master Spring Data.
The privilegied option is to use Native Rest client of ES 5.0.

In waiting you can also use my Implementation : https://github.com/VanRoy/spring-data-jest
I have planned the next release for the next week.

Julien.

Any news here? Can someone please share their thoughts on this? Will it ever make it into spring-data-elasticsearch? What is the stopper here - elasticsearch 6.0.0-alpha1 is already out.

It's a real bummer that there is no progress here. A missing native support is almost a bug, especially with some providers like AWS not offering access to the node ports of the cluster.

Third party solutions are suitable as aworkaround but are not updated frequently enough.

Tobias Günther I think DATAES-407 should cover what you are looking for.

Like Don Wellington said, the ElasicsearcRestTemplate implemented in DATAES-407 uses the RestClient of Elasticsearch. So I think this issue can be closed - the PR it references is closed since May 2016.

I resolved this as won't do, as the functionality is contained in the `ElasticsearchRestClient`, so this change is not needed any more.