如何打造基于s3cmd的短地址服务

这篇文章主要讲解了"如何打造基于s3cmd的短地址服务"，文中的讲解内容简单清晰，易于学习与理解，下面请大家跟着小编的思路慢慢深入，一起来研究和学习"如何打造基于s3cmd的短地址服务"吧！

需求描述

有线上项目使用S3去发布移动端APP，因为需要做权限认证所以对应的Object无法开"public-read"权限，尝试使用Presign方式生成的签名URL存在以下问题

• 生成的签名URL地址包含敏感信息，会暴露bucket名称和accesskey，带来安全隐患。

• 生成的签名URL地址太长，影响用户体验。

• 生成的地址有效时间内可以任意访问，无法做到比较严格的反盗链。

解决方案

使用openresty实现对S3数据访问流程的封装，客户端的数据下载全部走openresty上面搭建的服务，该服务提供一次性的短地址生成服务(访问几次以后对应短地址失效)，具体流程如下:
1. 用户数据上传到S3，并设置相应Object权限为"public-read"，取得对应的对外访问URL。
2. 以之前生成的对外访问URL为基础，生成对应的短地址服务。
3. 客户端使用短地址进行访问，超出访问次数则无法访问。

如果想进一步提升安全性，可以将RGW和Openresty通过内网互联，数据上传全部走内部网络，外部访问走外网。

具体实现

1. Openresty安装

2.安装依赖lua库文件

默认openresty的lib路径为/usr/local/openresty/lualib
需要依赖两个lua文件，其中url.lua主要用于url地址的解析，redis_iresty.lua用于redis的连接

将下面的源码文件保存为 /usr/local/openresty/lualib/resty/url.luahttps://github.com/golgote/neturl/blob/master/lib/net/url.lua将下面的源码文件保存为 /usr/local/openresty/lualib/resty/redis_iresty.luahttps://gist.github.com/moonbingbing/9915c66346e8fddcefb5

3. RGW服务配置

root@demohost:/etc/openresty# cat /etc/ceph/ceph.conf...[client.radosgw.demo]     rgw dns name = s3.cephbook.com #S3 endpoint对应的域名     rgw frontends = "civetweb port=7480"     host = demohost     keyring = /etc/ceph/ceph.client.radosgw.keyring     log file = /home/ceph/log/radosgw.log

4. Openresty配置

root@demohost:/etc/openresty# cat /etc/openresty/nginx.conf...server {        listen       80;        server_name  ceph.vip; #短地址主机头设置        location = /url { #生成短地址入口             content_by_lua_file /etc/openresty/url.lua;         }         location / { #提供短地址对应的数据访问             proxy_http_version 1.1;             proxy_set_header Host $host;             access_by_lua_file /etc/openresty/access.lua;             proxy_pass http://127.0.0.1:7480; #对应后端的radosgw服务入口         }     }

生成短地址服务的源码如下

root@demohost:/etc/openresty# cat url.lualocal request_method = ngx.var.request_methodlocal url_ = require "resty.url" #对应之前的url库local redis = require "resty.redis_iresty" #对应之前的redis_iresty库local s3_endpoint = "s3.cephbook.com" #这里设置只允许生成与S3 endpoint相关的短地址local endpoint = s3_endpoint .. "$"local charset = {}for i = 48,  57 do table.insert(charset, string.char(i)) endfor i = 65,  90 do table.insert(charset, string.char(i)) endfor i = 97, 122 do table.insert(charset, string.char(i)) endfunction string.random(length)  local urandom = assert(io.open('/dev/urandom','rb'))  local a, b, c, d = urandom:read(4):byte(1,4)  urandom:close()  local seed = a*0x1000000 + b*0x10000 + c *0x100 + d  math.randomseed(seed)  if length > 0 then    return string.random(length - 1) .. charset[math.random(1, #charset)]  else    return ""  endendfunction genera_url(red,url_id,host,path)    -- counts表示短地址最大访问次数，current表示当前次数    ok, err = red:hmset(url_id,'host',host,'uri',path,'counts',3,'current',1)    if not ok then        ngx.status = ngx.HTTP_METHOD_NOT_IMPLEMENTED        ngx.say("failed to hmset: ", err)        return ngx.exit(ngx.HTTP_METHOD_NOT_IMPLEMENTED)    end    -- 设置短地址记录最多能够在redis里面存储的时长，避免资源耗尽    ok, err = red:expire(url_id,3600)     if not ok then        ngx.status = ngx.HTTP_METHOD_NOT_IMPLEMENTED        ngx.say("failed to set expire: ", err)        return ngx.exit(ngx.HTTP_METHOD_NOT_IMPLEMENTED)    endendfunction get_info_by_id(red,url_id)    ok, err = red:hgetall(url_id)    if not ok then        ngx.status = ngx.HTTP_SERVICE_UNAVAILABLE        ngx.say("failed to getall : ", err)        return ngx.exit(ngx.HTTP_SERVICE_UNAVAILABLE)    end    local h = red:array_to_hash(ok)    return hendif request_method == "POST" then    ngx.req.read_body()    local data = ngx.req.get_body_data()    local u = url_.parse(data)    local hostname_check, hostname_err = ngx.re.match(u.host,endpoint)    if not hostname_check then        ngx.status = ngx.HTTP_BAD_REQUEST        ngx.say("not a available s3_endpoint")        ngx.exit(ngx.HTTP_BAD_REQUEST)    end    if u.host then        if string.len(u.path) > 1  then            local url_id = string.random(7)            local red = redis:new()            genera_url(red,url_id,u.host,u.path)            local h = get_info_by_id(red,url_id)            ngx.status = ngx.HTTP_CREATED            ngx.say("ShortURL: ",ngx.var.scheme,"://",ngx.var.host,"/",url_id)            ngx.say("host: ", h.host)            ngx.say("uri: ", h.uri)            ngx.say("counts: ", h.counts)            ngx.say("current: ", h.current)            ngx.exit(ngx.HTTP_CREATED)        else            ngx.status = ngx.HTTP_BAD_REQUEST            ngx.say("path err")            ngx.exit(ngx.HTTP_BAD_REQUEST)        end    else        ngx.status = ngx.HTTP_BAD_REQUEST        ngx.say("not a available url")        ngx.exit(ngx.HTTP_BAD_REQUEST)    endend

访问短地址数据服务的源码如下

root@demohost:/etc/openresty# cat /etc/openresty/access.lualocal redis = require "resty.redis_iresty"local red = redis:new()local uri =  ngx.var.urilocal url_id = string.sub(uri,2,string.len(uri))res, err = red:hgetall(url_id)if not res then    ngx.exit(ngx.HTTP_NOT_FOUND)else    local h = red:array_to_hash(res)    -- 超出访问次数则拒绝访问，如果考虑资源占用可以自己加上删除对应redis记录操作    if h.counts < h.current then        ngx.status = ngx.HTTP_GONE        ngx.say("current=",h.current," > counts=",h.counts)        ngx.exit(ngx.HTTP_GONE)    end    ngx.req.set_header("host", h.host)    ngx.req.set_uri(h.uri, false)    ok, err = red:hincrby(url_id,'current',1)    if not ok then        ngx.status = ngx.HTTP_METHOD_NOT_IMPLEMENTED        ngx.say("incrby key failed: ", err)        ngx.exit(ngx.HTTP_METHOD_NOT_IMPLEMENTED)        return    endend

https://github.com/openresty/lua-nginx-module#http-status-constants

5. 使用流程

上传数据

使用s3cmd或者是其他方式上传文件，并设置对应的Object访问权限为"public-read"。

root@demohost:/tmp# s3cmd put myfile s3://demo --acl-public'myfile' -> 's3://demo/myfile'  [1 of 1] 21577 of 21577   100% in    0s   227.71 kB/s  done'myfile' -> 's3://demo/myfile'  [1 of 1] 21577 of 21577   100% in    0s   203.11 kB/s  donePublic URL of the object is: http://demo.s3.cephbook.com/myfile

生成短地址

root@demohost:/tmp# curl ceph.vip/url  -d "http://demo.s3.cephbook.com/myfile" -v* Hostname was NOT found in DNS cache* Connected to ceph.vip  port 80 (#0)> POST /url HTTP/1.1> User-Agent: curl/7.38.0> Host: ceph.vip> Accept: */*> Content-Length: 29> Content-Type: application/x-www-form-urlencoded>* upload completely sent off: 29 out of 29 bytes< HTTP/1.1 201 Created* Server openresty/1.11.2.5 is not blacklisted< Server: openresty/1.11.2.5< Date: Wed, 15 Nov 2017 09:56:14 GMT< Content-Type: application/octet-stream< Transfer-Encoding: chunked< Connection: keep-alive
测试访问
#访问3次root@demohost:/tmp# curl http://ceph.vip/Vo3t5Ex -v...root@demohost:/tmp# curl http://ceph.vip/Vo3t5Ex -v...root@demohost:/tmp# curl http://ceph.vip/Vo3t5Ex -v...#第四次以后就不能访问了root@demohost:/tmp# curl http://ceph.vip/Vo3t5Ex -v* Hostname was NOT found in DNS cache*   Trying ceph.vip...* Connected to ceph.vip (ceph.vip) port 80 (#0)> GET /Vo3t5Ex HTTP/1.1> User-Agent: curl/7.38.0> Host: ceph.vip> Accept: */*>< HTTP/1.1 410 Gone* Server openresty/1.11.2.5 is not blacklisted< Server: openresty/1.11.2.5< Date: Wed, 15 Nov 2017 10:00:46 GMT< Content-Type: application/octet-stream< Transfer-Encoding: chunked< Connection: keep-alive counts=3* Connection #0 to host ceph.vip left intact
感谢各位的阅读，以上就是"如何打造基于s3cmd的短地址服务"的内容了，经过本文的学习后，相信大家对如何打造基于s3cmd的短地址服务这一问题有了更深刻的体会，具体使用情况还需要大家实践验证。这里是，小编将为大家推送更多相关知识点的文章，欢迎关注！